Nov 04 00:54:54.398196 ip-10-0-128-129 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 04 00:54:54.398208 ip-10-0-128-129 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 04 00:54:54.398217 ip-10-0-128-129 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 04 00:54:54.398533 ip-10-0-128-129 systemd[1]: Failed to start Kubernetes Kubelet. Nov 04 00:55:04.597026 ip-10-0-128-129 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 04 00:55:04.597041 ip-10-0-128-129 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 1965753a7ae643768604d00adf320a03 -- Nov 04 00:57:22.977188 ip-10-0-128-129 systemd[1]: Starting Kubernetes Kubelet... Nov 04 00:57:23.337477 ip-10-0-128-129 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 00:57:23.337477 ip-10-0-128-129 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 04 00:57:23.337477 ip-10-0-128-129 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 00:57:23.337477 ip-10-0-128-129 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 00:57:23.341342 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341264 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:23.341342 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341342 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:23.341406 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341347 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:23.341406 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341350 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:23.341406 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341353 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:23.341406 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341356 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:23.341406 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341359 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:23.341406 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341361 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:23.341406 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341364 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:23.341406 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341367 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:23.341406 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341369 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:23.341406 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341373 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:23.341406 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341375 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:23.341406 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341378 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:23.341406 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341380 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:23.341406 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341383 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:23.341406 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341395 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:23.341406 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341398 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:23.341406 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341400 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:23.341406 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341403 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:23.341406 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341406 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:23.341406 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341409 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:23.341878 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341412 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:23.341878 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341415 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:23.341878 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341417 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:23.341878 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341420 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:23.341878 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341423 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:23.341878 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341425 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:23.341878 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341428 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:23.341878 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341431 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:23.341878 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341433 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:23.341878 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341435 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:23.341878 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341438 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:23.341878 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341440 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:23.341878 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341443 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:23.341878 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341445 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:23.341878 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341447 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:23.341878 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341450 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:23.341878 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341452 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:23.341878 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341455 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:23.341878 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341457 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:23.341878 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341459 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:23.342359 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341462 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:23.342359 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341465 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:23.342359 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341468 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:23.342359 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341470 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:23.342359 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341473 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:23.342359 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341475 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:23.342359 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341478 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:23.342359 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341480 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:23.342359 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341483 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:23.342359 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341485 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:23.342359 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341487 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:23.342359 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341490 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:23.342359 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341493 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:23.342359 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341496 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:23.342359 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341499 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:23.342359 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341503 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:23.342359 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341507 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:23.342359 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341510 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:23.342359 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341513 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:23.342817 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341515 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:23.342817 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341518 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:23.342817 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341520 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:23.342817 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341523 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:23.342817 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341526 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:23.342817 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341528 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:23.342817 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341531 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:23.342817 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341533 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:23.342817 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341536 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:23.342817 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341539 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:23.342817 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341542 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:23.342817 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341544 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:23.342817 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341547 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:23.342817 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341549 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:23.342817 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341552 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:23.342817 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341554 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:23.342817 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341557 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:23.342817 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341559 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:23.342817 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341562 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:23.343286 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341565 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:23.343286 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341567 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:23.343286 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341570 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:23.343286 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341572 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:23.343286 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341574 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:23.343286 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341578 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:23.343286 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341580 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:23.343286 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341583 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:23.343286 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341587 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:23.343286 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341591 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:23.343286 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341593 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:23.343286 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341597 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:23.343286 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341600 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:23.343286 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341602 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:23.343286 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341605 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:23.343286 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341609 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:23.343286 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341612 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:23.343286 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341614 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:23.343286 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341616 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:23.343286 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341619 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:23.343764 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341621 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:23.343764 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341624 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:23.343764 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341626 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:23.343764 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341628 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:23.343764 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341631 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:23.343764 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.341633 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:23.344037 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.343979 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:23.344037 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.343990 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:23.344037 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.343995 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:23.344037 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344000 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:23.344037 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344005 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:23.344037 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344011 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:23.344037 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344015 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:23.344037 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344020 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:23.344037 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344025 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:23.344037 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344029 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:23.344037 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344033 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:23.344037 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344038 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:23.344037 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344043 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:23.344037 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344048 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:23.344417 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344053 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:23.344417 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344057 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:23.344417 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344061 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:23.344417 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344066 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:23.344417 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344071 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:23.344417 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344075 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:23.344417 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344080 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:23.344417 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344083 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:23.344417 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344085 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:23.344417 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344088 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:23.344417 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344091 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:23.344417 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344093 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:23.344417 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344096 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:23.344417 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344098 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:23.344417 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344100 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:23.344417 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344103 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:23.344417 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344106 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:23.344417 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344108 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:23.344417 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344110 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:23.344417 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344113 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:23.344898 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344115 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:23.344898 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344118 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:23.344898 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344120 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:23.344898 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344122 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:23.344898 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344125 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:23.344898 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344128 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:23.344898 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344131 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:23.344898 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344133 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:23.344898 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344136 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:23.344898 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344139 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:23.344898 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344141 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:23.344898 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344143 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:23.344898 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344146 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:23.344898 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344149 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:23.344898 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344152 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:23.344898 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344155 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:23.344898 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344157 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:23.344898 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344160 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:23.344898 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344163 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:23.345398 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344165 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:23.345398 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344167 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:23.345398 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344170 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:23.345398 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344172 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:23.345398 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344175 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:23.345398 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344177 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:23.345398 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344180 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:23.345398 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344182 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:23.345398 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344185 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:23.345398 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344187 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:23.345398 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344189 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:23.345398 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344192 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:23.345398 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344194 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:23.345398 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344197 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:23.345398 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344199 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:23.345398 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344201 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:23.345398 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344204 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:23.345398 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344206 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:23.345398 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344210 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:23.345398 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344212 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:23.345876 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344215 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:23.345876 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344217 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:23.345876 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344220 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:23.345876 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344222 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:23.345876 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344226 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:23.345876 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344229 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:23.345876 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344232 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:23.345876 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344234 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:23.345876 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344237 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:23.345876 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344241 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:23.345876 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344245 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:23.345876 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344248 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:23.345876 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344251 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:23.345876 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344254 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:23.345876 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344256 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:23.345876 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344259 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:23.345876 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344262 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:23.345876 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344264 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:23.345876 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344267 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:23.346371 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344269 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:23.346371 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344271 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:23.346371 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344274 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:23.346371 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344276 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:23.346371 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344278 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:23.346371 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344282 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:23.346371 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344286 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:23.346371 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344288 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:23.346371 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344291 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:23.346371 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344293 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:23.346371 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344295 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:23.346371 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344298 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:23.346371 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344302 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:23.346371 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344305 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:23.346371 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344788 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:23.346371 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344793 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:23.346371 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344796 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:23.346371 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344799 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:23.346371 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344802 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:23.346822 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344805 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:23.346822 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344808 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:23.346822 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344810 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:23.346822 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344813 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:23.346822 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344815 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:23.346822 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344818 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:23.346822 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344821 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:23.346822 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344823 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:23.346822 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344825 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:23.346822 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344828 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:23.346822 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344832 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:23.346822 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344837 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:23.346822 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344840 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:23.346822 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344842 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:23.346822 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344845 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:23.346822 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344848 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:23.346822 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344850 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:23.346822 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344852 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:23.346822 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344855 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:23.346822 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344857 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:23.347331 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344860 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:23.347331 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344862 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:23.347331 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344865 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:23.347331 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344867 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:23.347331 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344869 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:23.347331 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344872 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:23.347331 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344875 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:23.347331 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344878 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:23.347331 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344881 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:23.347331 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344884 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:23.347331 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344888 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:23.347331 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344891 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:23.347331 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344894 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:23.347331 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344896 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:23.347331 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344899 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:23.347331 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344902 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:23.347331 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344904 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:23.347331 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344907 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:23.347331 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344910 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:23.347824 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344913 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:23.347824 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344915 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:23.347824 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344917 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:23.347824 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344935 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:23.347824 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344938 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:23.347824 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344940 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:23.347824 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344943 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:23.347824 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344945 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:23.347824 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344948 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:23.347824 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344950 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:23.347824 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344953 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:23.347824 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344955 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:23.347824 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344958 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:23.347824 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344960 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:23.347824 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344962 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:23.347824 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344965 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:23.347824 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344967 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:23.347824 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344970 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:23.347824 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344972 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:23.348395 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344976 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:23.348395 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344979 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:23.348395 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344982 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:23.348395 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344985 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:23.348395 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344987 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:23.348395 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344990 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:23.348395 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344995 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:23.348395 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.344997 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:23.348395 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345001 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:23.348395 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345003 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:23.348395 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345006 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:23.348395 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345009 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:23.348395 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345011 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:23.348395 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345014 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:23.348395 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345016 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:23.348395 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345019 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:23.348395 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345021 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:23.348395 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345024 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:23.348395 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345026 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:23.348847 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345029 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:23.348847 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345031 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:23.348847 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345034 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:23.348847 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345036 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:23.348847 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345039 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:23.348847 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345041 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:23.348847 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345044 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:23.348847 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345046 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:23.348847 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345048 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:23.348847 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345051 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:23.348847 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345053 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:23.348847 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345056 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:23.348847 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345058 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:23.348847 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345062 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:23.348847 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345065 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:23.348847 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345068 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:23.348847 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345070 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:23.348847 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345073 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:23.348847 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345075 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:23.348847 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345079 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:23.349396 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345082 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:23.349396 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345084 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:23.349396 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345087 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:23.349396 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345089 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:23.349396 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345184 2577 flags.go:64] FLAG: --address="0.0.0.0" Nov 04 00:57:23.349396 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345191 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 04 00:57:23.349396 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345198 2577 flags.go:64] FLAG: --anonymous-auth="true" Nov 04 00:57:23.349396 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345202 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 04 00:57:23.349396 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345206 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 04 00:57:23.349396 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345209 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 04 00:57:23.349396 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345213 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 04 00:57:23.349396 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345217 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 04 00:57:23.349396 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345221 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 04 00:57:23.349396 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345224 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 04 00:57:23.349396 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345227 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 04 00:57:23.349396 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345230 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 04 00:57:23.349396 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345233 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 04 00:57:23.349396 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345236 2577 flags.go:64] FLAG: --cgroup-root="" Nov 04 00:57:23.349396 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345239 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 04 00:57:23.349396 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345241 2577 flags.go:64] FLAG: --client-ca-file="" Nov 04 00:57:23.349396 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345244 2577 flags.go:64] FLAG: --cloud-provider="external" Nov 04 00:57:23.349396 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345247 2577 flags.go:64] FLAG: --cluster-dns="[]" Nov 04 00:57:23.349396 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345251 2577 flags.go:64] FLAG: --cluster-domain="" Nov 04 00:57:23.349396 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345254 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 04 00:57:23.349988 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345258 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 04 00:57:23.349988 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345261 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 04 00:57:23.349988 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345264 2577 flags.go:64] FLAG: --container-log-max-files="5" Nov 04 00:57:23.349988 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345269 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 04 00:57:23.349988 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345272 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 04 00:57:23.349988 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345276 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 04 00:57:23.349988 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345279 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 04 00:57:23.349988 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345284 2577 flags.go:64] FLAG: --contention-profiling="false" Nov 04 00:57:23.349988 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345287 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 04 00:57:23.349988 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345290 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 04 00:57:23.349988 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345293 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 04 00:57:23.349988 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345296 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 04 00:57:23.349988 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345300 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 04 00:57:23.349988 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345303 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 04 00:57:23.349988 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345306 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 04 00:57:23.349988 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345308 2577 flags.go:64] FLAG: --enable-load-reader="false" Nov 04 00:57:23.349988 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345311 2577 flags.go:64] FLAG: --enable-server="true" Nov 04 00:57:23.349988 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345314 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 04 00:57:23.349988 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345318 2577 flags.go:64] FLAG: --event-burst="100" Nov 04 00:57:23.349988 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345321 2577 flags.go:64] FLAG: --event-qps="50" Nov 04 00:57:23.349988 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345324 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 04 00:57:23.349988 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345327 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 04 00:57:23.349988 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345330 2577 flags.go:64] FLAG: --eviction-hard="" Nov 04 00:57:23.349988 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345334 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 04 00:57:23.350576 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345337 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 04 00:57:23.350576 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345340 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 04 00:57:23.350576 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345343 2577 flags.go:64] FLAG: --eviction-soft="" Nov 04 00:57:23.350576 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345346 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 04 00:57:23.350576 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345349 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 04 00:57:23.350576 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345352 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 04 00:57:23.350576 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345355 2577 flags.go:64] FLAG: --experimental-mounter-path="" Nov 04 00:57:23.350576 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345358 2577 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 04 00:57:23.350576 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345361 2577 flags.go:64] FLAG: --fail-swap-on="true" Nov 04 00:57:23.350576 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345365 2577 flags.go:64] FLAG: --feature-gates="" Nov 04 00:57:23.350576 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345369 2577 flags.go:64] FLAG: --file-check-frequency="20s" Nov 04 00:57:23.350576 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345372 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 04 00:57:23.350576 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345375 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 04 00:57:23.350576 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345379 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 04 00:57:23.350576 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345382 2577 flags.go:64] FLAG: --healthz-port="10248" Nov 04 00:57:23.350576 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345386 2577 flags.go:64] FLAG: --help="false" Nov 04 00:57:23.350576 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345389 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:57:23.350576 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345392 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 04 00:57:23.350576 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345395 2577 flags.go:64] FLAG: --http-check-frequency="20s" Nov 04 00:57:23.350576 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345398 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 04 00:57:23.350576 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345401 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 04 00:57:23.350576 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345405 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 04 00:57:23.350576 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345408 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 04 00:57:23.351206 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345410 2577 flags.go:64] FLAG: --image-service-endpoint="" Nov 04 00:57:23.351206 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345413 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 04 00:57:23.351206 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345416 2577 flags.go:64] FLAG: --kube-api-burst="100" Nov 04 00:57:23.351206 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345419 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 04 00:57:23.351206 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345422 2577 flags.go:64] FLAG: --kube-api-qps="50" Nov 04 00:57:23.351206 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345425 2577 flags.go:64] FLAG: --kube-reserved="" Nov 04 00:57:23.351206 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345428 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 04 00:57:23.351206 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345431 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 04 00:57:23.351206 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345434 2577 flags.go:64] FLAG: --kubelet-cgroups="" Nov 04 00:57:23.351206 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345436 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 04 00:57:23.351206 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345439 2577 flags.go:64] FLAG: --lock-file="" Nov 04 00:57:23.351206 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345442 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 04 00:57:23.351206 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345445 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 04 00:57:23.351206 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345448 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 04 00:57:23.351206 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345453 2577 flags.go:64] FLAG: --log-json-split-stream="false" Nov 04 00:57:23.351206 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345456 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 04 00:57:23.351206 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345459 2577 flags.go:64] FLAG: --log-text-split-stream="false" Nov 04 00:57:23.351206 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345462 2577 flags.go:64] FLAG: --logging-format="text" Nov 04 00:57:23.351206 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345466 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 04 00:57:23.351206 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345469 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 04 00:57:23.351206 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345472 2577 flags.go:64] FLAG: --manifest-url="" Nov 04 00:57:23.351206 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345475 2577 flags.go:64] FLAG: --manifest-url-header="" Nov 04 00:57:23.351206 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345480 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 04 00:57:23.351206 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345483 2577 flags.go:64] FLAG: --max-open-files="1000000" Nov 04 00:57:23.351206 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345488 2577 flags.go:64] FLAG: --max-pods="110" Nov 04 00:57:23.351803 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345491 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 04 00:57:23.351803 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345494 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 04 00:57:23.351803 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345497 2577 flags.go:64] FLAG: --memory-manager-policy="None" Nov 04 00:57:23.351803 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345500 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 04 00:57:23.351803 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345503 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 04 00:57:23.351803 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345506 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 04 00:57:23.351803 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345509 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 04 00:57:23.351803 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345515 2577 flags.go:64] FLAG: --node-status-max-images="50" Nov 04 00:57:23.351803 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345519 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 04 00:57:23.351803 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345522 2577 flags.go:64] FLAG: --oom-score-adj="-999" Nov 04 00:57:23.351803 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345525 2577 flags.go:64] FLAG: --pod-cidr="" Nov 04 00:57:23.351803 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345528 2577 flags.go:64] FLAG: --pod-manifest-path="" Nov 04 00:57:23.351803 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345531 2577 flags.go:64] FLAG: --pod-max-pids="-1" Nov 04 00:57:23.351803 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345534 2577 flags.go:64] FLAG: --pods-per-core="0" Nov 04 00:57:23.351803 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345537 2577 flags.go:64] FLAG: --port="10250" Nov 04 00:57:23.351803 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345540 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 04 00:57:23.351803 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345542 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-000c8fb1108dc6043" Nov 04 00:57:23.351803 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345545 2577 flags.go:64] FLAG: --qos-reserved="" Nov 04 00:57:23.351803 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345548 2577 flags.go:64] FLAG: --read-only-port="10255" Nov 04 00:57:23.351803 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345551 2577 flags.go:64] FLAG: --register-node="true" Nov 04 00:57:23.351803 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345554 2577 flags.go:64] FLAG: --register-with-taints="" Nov 04 00:57:23.351803 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345557 2577 flags.go:64] FLAG: --registry-burst="10" Nov 04 00:57:23.351803 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345560 2577 flags.go:64] FLAG: --registry-qps="5" Nov 04 00:57:23.351803 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345562 2577 flags.go:64] FLAG: --reserved-cpus="" Nov 04 00:57:23.351803 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345565 2577 flags.go:64] FLAG: --reserved-memory="" Nov 04 00:57:23.351803 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345570 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 04 00:57:23.352474 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345573 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 04 00:57:23.352474 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345576 2577 flags.go:64] FLAG: --rotate-certificates="false" Nov 04 00:57:23.352474 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345579 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 04 00:57:23.352474 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345582 2577 flags.go:64] FLAG: --runonce="false" Nov 04 00:57:23.352474 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345585 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 04 00:57:23.352474 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345590 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 04 00:57:23.352474 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345593 2577 flags.go:64] FLAG: --seccomp-default="false" Nov 04 00:57:23.352474 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345595 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 04 00:57:23.352474 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345598 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 04 00:57:23.352474 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345601 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 04 00:57:23.352474 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345604 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 04 00:57:23.352474 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345607 2577 flags.go:64] FLAG: --storage-driver-password="root" Nov 04 00:57:23.352474 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345610 2577 flags.go:64] FLAG: --storage-driver-secure="false" Nov 04 00:57:23.352474 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345613 2577 flags.go:64] FLAG: --storage-driver-table="stats" Nov 04 00:57:23.352474 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345616 2577 flags.go:64] FLAG: --storage-driver-user="root" Nov 04 00:57:23.352474 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345619 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 04 00:57:23.352474 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345622 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 04 00:57:23.352474 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345624 2577 flags.go:64] FLAG: --system-cgroups="" Nov 04 00:57:23.352474 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345627 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 04 00:57:23.352474 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345632 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 04 00:57:23.352474 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345635 2577 flags.go:64] FLAG: --tls-cert-file="" Nov 04 00:57:23.352474 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345638 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 04 00:57:23.352474 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345642 2577 flags.go:64] FLAG: --tls-min-version="" Nov 04 00:57:23.352474 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345644 2577 flags.go:64] FLAG: --tls-private-key-file="" Nov 04 00:57:23.352474 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345647 2577 flags.go:64] FLAG: --topology-manager-policy="none" Nov 04 00:57:23.353072 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345650 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 04 00:57:23.353072 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345653 2577 flags.go:64] FLAG: --topology-manager-scope="container" Nov 04 00:57:23.353072 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345655 2577 flags.go:64] FLAG: --v="2" Nov 04 00:57:23.353072 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345660 2577 flags.go:64] FLAG: --version="false" Nov 04 00:57:23.353072 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345664 2577 flags.go:64] FLAG: --vmodule="" Nov 04 00:57:23.353072 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345668 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 04 00:57:23.353072 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.345672 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 04 00:57:23.353072 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345801 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:23.353072 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345804 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:23.353072 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345808 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:23.353072 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345810 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:23.353072 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345813 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:23.353072 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345817 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:23.353072 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345821 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:23.353072 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345825 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:23.353072 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345828 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:23.353072 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345831 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:23.353072 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345834 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:23.353072 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345837 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:23.353072 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345839 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:23.353072 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345842 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:23.353570 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345845 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:23.353570 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345848 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:23.353570 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345850 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:23.353570 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345852 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:23.353570 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345855 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:23.353570 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345858 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:23.353570 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345860 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:23.353570 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345863 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:23.353570 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345865 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:23.353570 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345868 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:23.353570 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345870 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:23.353570 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345873 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:23.353570 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345875 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:23.353570 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345878 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:23.353570 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345881 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:23.353570 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345883 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:23.353570 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345886 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:23.353570 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345890 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:23.353570 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345894 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:23.353570 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345896 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:23.354060 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345899 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:23.354060 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345901 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:23.354060 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345904 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:23.354060 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345908 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:23.354060 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345911 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:23.354060 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345914 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:23.354060 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345916 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:23.354060 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345918 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:23.354060 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345936 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:23.354060 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345939 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:23.354060 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345941 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:23.354060 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345944 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:23.354060 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345947 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:23.354060 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345949 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:23.354060 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345951 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:23.354060 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345954 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:23.354060 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345957 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:23.354060 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345959 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:23.354060 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345962 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:23.354060 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345964 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:23.354533 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345967 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:23.354533 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345969 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:23.354533 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345972 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:23.354533 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345974 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:23.354533 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345977 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:23.354533 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345979 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:23.354533 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345982 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:23.354533 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345984 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:23.354533 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345987 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:23.354533 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345991 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:23.354533 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345994 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:23.354533 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345997 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:23.354533 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.345999 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:23.354533 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346002 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:23.354533 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346004 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:23.354533 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346007 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:23.354533 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346009 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:23.354533 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346012 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:23.354533 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346015 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:23.354533 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346018 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:23.355122 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346020 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:23.355122 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346023 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:23.355122 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346025 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:23.355122 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346028 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:23.355122 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346031 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:23.355122 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346033 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:23.355122 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346036 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:23.355122 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346038 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:23.355122 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346041 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:23.355122 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346043 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:23.355122 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346046 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:23.355122 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346048 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:23.355122 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346051 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:23.355122 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346053 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:23.355122 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346056 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:23.355122 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346058 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:23.355122 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346062 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:23.355122 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346065 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:23.355122 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346068 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:23.355567 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346070 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:23.355567 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346072 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:23.355567 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346076 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:23.355567 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346079 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:23.355567 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346081 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:23.355567 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346084 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:23.355567 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346087 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:23.355567 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346089 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:23.355567 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346092 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:23.355567 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346095 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:23.355567 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346097 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:23.355567 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346100 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:23.355567 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346102 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:23.355567 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.346130 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 00:57:23.355567 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346165 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:23.355567 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346169 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:23.355954 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346172 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:23.355954 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346175 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:23.355954 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346178 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:23.355954 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346181 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:23.355954 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346183 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:23.355954 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346186 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:23.355954 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346188 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:23.355954 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346191 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:23.355954 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346193 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:23.355954 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346195 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:23.355954 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346198 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:23.355954 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346200 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:23.355954 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346203 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:23.355954 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346205 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:23.355954 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346208 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:23.355954 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346210 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:23.355954 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346213 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:23.355954 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346215 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:23.355954 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346218 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:23.355954 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346221 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:23.356429 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346224 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:23.356429 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346227 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:23.356429 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346229 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:23.356429 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346232 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:23.356429 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346234 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:23.356429 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346237 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:23.356429 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346240 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:23.356429 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346242 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:23.356429 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346245 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:23.356429 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346248 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:23.356429 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346251 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:23.356429 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346256 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:23.356429 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346258 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:23.356429 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346261 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:23.356429 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346264 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:23.356429 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346267 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:23.356429 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346270 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:23.356429 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346273 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:23.356429 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346276 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:23.356871 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346279 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:23.356871 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346281 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:23.356871 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346284 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:23.356871 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346286 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:23.356871 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346289 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:23.356871 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346291 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:23.356871 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346293 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:23.356871 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346296 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:23.356871 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346298 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:23.356871 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346301 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:23.356871 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346304 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:23.356871 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346306 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:23.356871 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346312 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:23.356871 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346316 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:23.356871 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346319 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:23.356871 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346324 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:23.356871 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346327 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:23.356871 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346330 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:23.356871 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346333 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:23.357349 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346335 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:23.357349 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346338 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:23.357349 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346340 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:23.357349 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346343 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:23.357349 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346346 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:23.357349 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346348 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:23.357349 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346351 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:23.357349 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346353 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:23.357349 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346355 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:23.357349 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346358 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:23.357349 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346360 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:23.357349 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346363 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:23.357349 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346365 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:23.357349 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346368 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:23.357349 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346370 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:23.357349 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346373 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:23.357349 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346375 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:23.357349 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346378 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:23.357349 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346381 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:23.357349 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346384 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:23.357812 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346386 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:23.357812 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346389 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:23.357812 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346391 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:23.357812 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346394 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:23.357812 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346396 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:23.357812 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346399 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:23.357812 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346402 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:23.357812 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346404 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:23.357812 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346408 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:23.357812 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346411 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:23.357812 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346413 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:23.357812 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346416 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:23.357812 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346418 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:23.357812 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346420 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:23.357812 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346423 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:23.357812 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346426 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:23.357812 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346428 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:23.357812 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346431 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:23.357812 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346433 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:23.357812 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346435 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:23.358334 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346438 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:23.358334 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346440 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:23.358334 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346443 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:23.358334 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346445 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:23.358334 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346448 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:23.358334 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.346450 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:23.358334 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.352947 2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 04 00:57:23.358334 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.352960 2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 04 00:57:23.358334 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.352973 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 04 00:57:23.358334 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.352978 2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 04 00:57:23.358334 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353035 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:23.358334 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353039 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:23.358334 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353043 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:23.358334 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353047 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:23.358334 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353050 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:23.358334 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353053 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:23.358334 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353056 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:23.358334 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353059 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:23.358334 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353061 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:23.358808 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353064 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:23.358808 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353066 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:23.358808 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353069 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:23.358808 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353071 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:23.358808 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353074 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:23.358808 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353077 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:23.358808 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353080 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:23.358808 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353082 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:23.358808 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353098 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:23.358808 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353103 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:23.358808 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353106 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:23.358808 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353109 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:23.358808 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353112 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:23.358808 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353115 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:23.358808 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353117 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:23.358808 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353120 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:23.358808 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353122 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:23.358808 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353125 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:23.358808 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353127 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:23.359287 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353130 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:23.359287 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353133 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:23.359287 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353136 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:23.359287 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353138 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:23.359287 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353141 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:23.359287 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353143 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:23.359287 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353145 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:23.359287 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353148 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:23.359287 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353150 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:23.359287 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353153 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:23.359287 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353155 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:23.359287 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353158 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:23.359287 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353161 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:23.359287 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353163 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:23.359287 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353165 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:23.359287 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353168 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:23.359287 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353170 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:23.359287 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353172 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:23.359287 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353175 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:23.359287 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353177 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:23.359287 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353180 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:23.359775 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353182 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:23.359775 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353185 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:23.359775 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353188 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:23.359775 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353190 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:23.359775 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353193 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:23.359775 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353195 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:23.359775 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353197 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:23.359775 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353200 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:23.359775 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353202 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:23.359775 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353205 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:23.359775 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353208 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:23.359775 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353212 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:23.359775 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353216 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:23.359775 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353220 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:23.359775 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353223 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:23.359775 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353226 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:23.359775 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353228 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:23.359775 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353231 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:23.359775 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353233 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:23.360281 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353236 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:23.360281 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353238 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:23.360281 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353241 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:23.360281 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353243 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:23.360281 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353245 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:23.360281 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353248 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:23.360281 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353250 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:23.360281 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353253 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:23.360281 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353255 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:23.360281 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353258 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:23.360281 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353260 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:23.360281 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353263 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:23.360281 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353266 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:23.360281 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353268 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:23.360281 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353270 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:23.360281 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353273 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:23.360281 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353276 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:23.360281 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353279 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:23.360281 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353281 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:23.360714 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353284 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:23.360714 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353286 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:23.360714 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353289 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:23.360714 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353292 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:23.360714 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353294 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:23.360714 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353296 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:23.360714 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353299 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:23.360714 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353302 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:23.360714 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353304 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:23.360714 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353307 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:23.360714 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353309 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:23.360714 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353312 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:23.360714 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353314 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:23.360714 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353316 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:23.360714 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353319 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:23.360714 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353321 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:23.360714 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353324 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:23.360714 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353326 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:23.360714 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353329 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:23.361161 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.353355 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 00:57:23.361161 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353489 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:23.361161 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353493 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:23.361161 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353496 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:23.361161 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353498 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:23.361161 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353501 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:23.361161 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353503 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:23.361161 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353506 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:23.361161 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353508 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:23.361161 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353510 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:23.361161 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353513 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:23.361161 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353516 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:23.361161 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353519 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:23.361161 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353521 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:23.361161 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353524 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:23.361510 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353526 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:23.361510 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353528 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:23.361510 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353531 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:23.361510 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353533 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:23.361510 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353536 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:23.361510 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353538 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:23.361510 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353541 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:23.361510 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353543 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:23.361510 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353546 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:23.361510 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353548 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:23.361510 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353550 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:23.361510 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353553 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:23.361510 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353556 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:23.361510 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353561 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:23.361510 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353563 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:23.361510 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353566 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:23.361510 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353568 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:23.361510 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353571 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:23.361510 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353574 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:23.361969 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353576 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:23.361969 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353579 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:23.361969 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353582 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:23.361969 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353584 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:23.361969 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353587 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:23.361969 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353590 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:23.361969 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353592 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:23.361969 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353595 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:23.361969 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353597 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:23.361969 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353601 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:23.361969 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353605 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:23.361969 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353607 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:23.361969 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353610 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:23.361969 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353613 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:23.361969 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353617 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:23.361969 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353619 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:23.361969 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353622 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:23.361969 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353625 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:23.361969 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353627 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:23.361969 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353630 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:23.362479 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353632 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:23.362479 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353634 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:23.362479 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353637 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:23.362479 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353639 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:23.362479 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353641 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:23.362479 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353644 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:23.362479 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353646 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:23.362479 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353649 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:23.362479 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353652 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:23.362479 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353654 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:23.362479 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353656 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:23.362479 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353659 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:23.362479 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353661 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:23.362479 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353663 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:23.362479 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353666 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:23.362479 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353668 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:23.362479 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353670 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:23.362479 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353673 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:23.362479 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353675 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:23.362960 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353678 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:23.362960 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353680 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:23.362960 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353682 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:23.362960 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353685 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:23.362960 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353688 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:23.362960 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353692 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:23.362960 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353695 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:23.362960 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353698 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:23.362960 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353700 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:23.362960 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353703 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:23.362960 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353705 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:23.362960 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353708 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:23.362960 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353710 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:23.362960 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353713 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:23.362960 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353715 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:23.362960 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353718 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:23.362960 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353720 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:23.362960 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353723 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:23.362960 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353725 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:23.362960 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353728 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:23.363433 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353730 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:23.363433 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353733 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:23.363433 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353735 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:23.363433 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353738 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:23.363433 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353740 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:23.363433 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353743 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:23.363433 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353745 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:23.363433 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353747 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:23.363433 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353750 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:23.363433 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353752 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:23.363433 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353755 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:23.363433 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353757 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:23.363433 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353760 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:23.363433 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353762 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:23.363433 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.353785 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 00:57:23.363433 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353816 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:23.363805 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353820 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:23.363805 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353823 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:23.363805 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353828 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:23.363805 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353830 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:23.363805 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353833 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:23.363805 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353836 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:23.363805 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353839 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:23.363805 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353842 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:23.363805 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353844 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:23.363805 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353848 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:23.363805 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353851 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:23.363805 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353854 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:23.363805 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353857 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:23.363805 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353860 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:23.363805 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353864 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:23.363805 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353867 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:23.363805 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353869 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:23.363805 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353872 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:23.363805 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353874 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:23.363805 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353877 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:23.364291 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353879 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:23.364291 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353882 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:23.364291 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353884 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:23.364291 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353887 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:23.364291 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353889 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:23.364291 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353892 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:23.364291 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353894 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:23.364291 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353897 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:23.364291 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353899 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:23.364291 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353902 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:23.364291 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353904 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:23.364291 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353906 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:23.364291 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353909 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:23.364291 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353911 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:23.364291 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353915 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:23.364291 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353918 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:23.364291 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353936 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:23.364291 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353940 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:23.364291 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353944 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:23.364743 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353947 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:23.364743 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353949 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:23.364743 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353952 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:23.364743 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353954 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:23.364743 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353957 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:23.364743 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353960 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:23.364743 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353962 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:23.364743 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353965 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:23.364743 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353967 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:23.364743 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353970 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:23.364743 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353972 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:23.364743 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353975 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:23.364743 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353977 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:23.364743 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353980 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:23.364743 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353983 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:23.364743 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353985 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:23.364743 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353988 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:23.364743 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353990 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:23.364743 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353993 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:23.364743 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353996 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:23.365486 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.353998 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:23.365486 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354001 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:23.365486 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354004 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:23.365486 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354006 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:23.365486 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354008 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:23.365486 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354011 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:23.365486 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354013 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:23.365486 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354017 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:23.365486 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354020 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:23.365486 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354023 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:23.365486 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354025 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:23.365486 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354029 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:23.365486 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354032 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:23.365486 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354034 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:23.365486 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354037 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:23.365486 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354039 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:23.365486 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354042 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:23.365486 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354044 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:23.365486 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354047 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:23.366002 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354049 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:23.366002 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354051 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:23.366002 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354054 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:23.366002 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354056 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:23.366002 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354059 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:23.366002 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354061 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:23.366002 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354064 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:23.366002 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354067 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:23.366002 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354069 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:23.366002 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354072 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:23.366002 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354074 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:23.366002 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354077 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:23.366002 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354079 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:23.366002 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354081 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:23.366002 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354084 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:23.366002 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354087 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:23.366002 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354089 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:23.366002 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354092 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:23.366002 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354094 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:23.366496 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354096 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:23.366496 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354100 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:23.366496 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354105 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:23.366496 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354109 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:23.366496 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354112 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:23.366496 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354118 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:23.366496 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354121 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:23.366496 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:23.354124 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:23.366496 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.354171 2577 server.go:957] "Client rotation is on, will bootstrap in background" Nov 04 00:57:23.366496 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.356206 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 04 00:57:23.366496 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.358533 2577 server.go:1014] "Starting client certificate rotation" Nov 04 00:57:23.366496 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.358630 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 00:57:23.366496 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.358672 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 00:57:23.366496 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.361187 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 00:57:23.367343 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.367326 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 00:57:23.379186 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.379168 2577 log.go:25] "Validated CRI v1 runtime API" Nov 04 00:57:23.382471 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.382458 2577 log.go:25] "Validated CRI v1 image API" Nov 04 00:57:23.383336 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.383303 2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 04 00:57:23.385699 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.385680 2577 fs.go:135] Filesystem UUIDs: map[780366c4-b376-4f0c-997d-64ac6d5ea20e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 7df6de30-b3da-4ecd-bcdf-8c2f3952a962:/dev/nvme0n1p3] Nov 04 00:57:23.385777 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.385699 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 04 00:57:23.386908 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.386880 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 04 00:57:23.391162 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.391050 2577 manager.go:217] Machine: {Timestamp:2026-11-04 00:57:23.389749263 +0000 UTC m=+0.301915770 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100268 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26de6bada58ada926556bddff64dd3 SystemUUID:ec26de6b-ada5-8ada-9265-56bddff64dd3 BootID:1965753a-7ae6-4376-8604-d00adf320a03 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d4:5b:94:15 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d4:5b:94:15 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e2:07:01:21:4e:83 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 04 00:57:23.391162 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.391162 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 04 00:57:23.391271 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.391263 2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 04 00:57:23.394073 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.394048 2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 04 00:57:23.394199 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.394074 2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-129.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 04 00:57:23.394246 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.394201 2577 topology_manager.go:143] "Creating topology manager with none policy" Nov 04 00:57:23.394246 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.394209 2577 container_manager_linux.go:311] "Creating device plugin manager" Nov 04 00:57:23.394246 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.394243 2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 00:57:23.394324 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.394261 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 00:57:23.394969 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.394959 2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 04 00:57:23.398412 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.398400 2577 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 04 00:57:23.398516 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.398508 2577 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 04 00:57:23.398552 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.398539 2577 kubelet.go:493] "Attempting to sync node with API server" Nov 04 00:57:23.398552 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.398549 2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 04 00:57:23.398684 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.398560 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 04 00:57:23.398684 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.398570 2577 kubelet.go:405] "Adding apiserver pod source" Nov 04 00:57:23.398684 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.398579 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 04 00:57:23.398900 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.398887 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 00:57:23.402934 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.401057 2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 04 00:57:23.405938 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.405906 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4n58v" Nov 04 00:57:23.406232 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.406221 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 00:57:23.406971 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.406959 2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 04 00:57:23.407005 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.406990 2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 04 00:57:23.407208 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.407198 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 04 00:57:23.407244 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.407214 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 04 00:57:23.407244 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.407221 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 04 00:57:23.407244 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.407233 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 04 00:57:23.407244 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.407242 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 04 00:57:23.407343 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.407248 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 04 00:57:23.407682 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.407673 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 04 00:57:23.407725 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.407684 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 04 00:57:23.407725 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.407691 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 04 00:57:23.407725 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.407697 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 04 00:57:23.407725 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.407706 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 04 00:57:23.407725 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.407714 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 04 00:57:23.408281 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.408271 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 04 00:57:23.408319 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.408282 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 04 00:57:23.412309 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.412296 2577 server.go:1263] "Started kubelet" Nov 04 00:57:23.412373 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.412349 2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 04 00:57:23.416241 ip-10-0-128-129 systemd[1]: Started Kubernetes Kubelet. Nov 04 00:57:23.416669 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.412413 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 04 00:57:23.417175 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.416854 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 04 00:57:23.417393 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.417363 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4n58v" Nov 04 00:57:23.418523 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.418407 2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 04 00:57:23.420716 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.420696 2577 server.go:317] "Adding debug handlers to kubelet server" Nov 04 00:57:23.421471 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:23.421437 2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 04 00:57:23.422132 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.422115 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-129.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 04 00:57:23.422458 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:23.422436 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-129.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 04 00:57:23.422458 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:23.422438 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 04 00:57:23.422940 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:23.421981 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-129.us-east-2.compute.internal.18e4b19deb147deb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-129.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-129.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:57:23.412270571 +0000 UTC m=+0.324437079,LastTimestamp:2026-11-04 00:57:23.412270571 +0000 UTC m=+0.324437079,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-129.us-east-2.compute.internal,}" Nov 04 00:57:23.424748 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.424729 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 04 00:57:23.424836 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.424732 2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 04 00:57:23.424836 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:23.424815 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-129.us-east-2.compute.internal\" not found" Nov 04 00:57:23.425386 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.425364 2577 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 04 00:57:23.425386 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.425387 2577 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 04 00:57:23.425525 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.425473 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 04 00:57:23.425525 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.425473 2577 reconstruct.go:97] "Volume reconstruction finished" Nov 04 00:57:23.425525 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.425496 2577 reconciler.go:29] "Reconciler: start to sync state" Nov 04 00:57:23.425765 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.425750 2577 factory.go:55] Registering systemd factory Nov 04 00:57:23.425765 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.425768 2577 factory.go:223] Registration of the systemd container factory successfully Nov 04 00:57:23.425981 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.425967 2577 factory.go:153] Registering CRI-O factory Nov 04 00:57:23.426041 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.425986 2577 factory.go:223] Registration of the crio container factory successfully Nov 04 00:57:23.426041 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.426033 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 04 00:57:23.426145 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.426052 2577 factory.go:103] Registering Raw factory Nov 04 00:57:23.426145 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.426068 2577 manager.go:1199] Started watching for new ooms in manager Nov 04 00:57:23.428615 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.428599 2577 manager.go:319] Starting recovery of all containers Nov 04 00:57:23.434625 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.434607 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:57:23.437032 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:23.437011 2577 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-129.us-east-2.compute.internal\" not found" node="ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:57:23.438374 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.438360 2577 manager.go:324] Recovery completed Nov 04 00:57:23.440484 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:23.440456 2577 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Nov 04 00:57:23.444069 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.444056 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:23.445717 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.445703 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-129.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:23.445772 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.445733 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-129.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:23.445772 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.445746 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-129.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:23.446297 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.446282 2577 cpu_manager.go:226] "Starting" policy="none" Nov 04 00:57:23.446297 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.446295 2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 04 00:57:23.446406 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.446309 2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 04 00:57:23.446406 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.446286 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:23.447110 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.447095 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-129.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:23.447168 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.447126 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-129.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:23.447168 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.447142 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-129.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:23.448441 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.448430 2577 policy_none.go:50] "Start" Nov 04 00:57:23.448487 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.448444 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 04 00:57:23.448487 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.448460 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 04 00:57:23.450580 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.450569 2577 policy_none.go:44] "Start" Nov 04 00:57:23.492504 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.492474 2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 04 00:57:23.494715 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.494700 2577 manager.go:341] "Starting Device Plugin manager" Nov 04 00:57:23.494802 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:23.494726 2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 04 00:57:23.494802 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.494734 2577 server.go:87] "Starting device plugin registration server" Nov 04 00:57:23.494979 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.494966 2577 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 04 00:57:23.495046 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.494979 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 04 00:57:23.495094 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.495066 2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 04 00:57:23.495164 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.495140 2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 04 00:57:23.495164 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.495162 2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 04 00:57:23.495532 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:23.495518 2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 04 00:57:23.495597 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:23.495559 2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-129.us-east-2.compute.internal\" not found" Nov 04 00:57:23.595106 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.595060 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:23.596887 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.595816 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-129.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:23.596887 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.595844 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-129.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:23.596887 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.595856 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-129.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:23.596887 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.595881 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:57:23.604817 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.604801 2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:57:23.604869 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:23.604824 2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-129.us-east-2.compute.internal\": node \"ip-10-0-128-129.us-east-2.compute.internal\" not found" Nov 04 00:57:23.626215 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:23.626190 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-129.us-east-2.compute.internal\" not found" Nov 04 00:57:23.726632 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:23.726613 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-129.us-east-2.compute.internal\" not found" Nov 04 00:57:23.763202 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.763179 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 04 00:57:23.764491 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.764477 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 04 00:57:23.764546 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.764498 2577 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 04 00:57:23.764546 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.764515 2577 kubelet.go:2521] "Starting kubelet main sync loop" Nov 04 00:57:23.764546 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:23.764543 2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 04 00:57:23.767793 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.767776 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:57:23.826721 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:23.826700 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-129.us-east-2.compute.internal\" not found" Nov 04 00:57:23.865481 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.865438 2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-129.us-east-2.compute.internal"] Nov 04 00:57:23.865542 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.865493 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:23.866292 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.866277 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-129.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:23.866373 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.866309 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-129.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:23.866373 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.866326 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-129.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:23.868316 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.868301 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:23.868449 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.868435 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:57:23.868496 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.868464 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:23.868993 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.868973 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-129.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:23.869072 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.869006 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-129.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:23.869072 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.869017 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-129.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:23.869072 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.868975 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-129.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:23.869166 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.869073 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-129.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:23.869166 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.869085 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-129.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:23.870917 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.870899 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:57:23.870990 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.870950 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:23.871650 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.871623 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-129.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:23.871736 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.871657 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-129.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:23.871736 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.871668 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-129.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:23.892021 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:23.892002 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-129.us-east-2.compute.internal\" not found" node="ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:57:23.893803 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:23.893787 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-129.us-east-2.compute.internal\" not found" node="ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:57:23.927616 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:23.927594 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-129.us-east-2.compute.internal\" not found" Nov 04 00:57:23.927700 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.927619 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/12bcfb3a1cdb54b757405d90e15ad22f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal\" (UID: \"12bcfb3a1cdb54b757405d90e15ad22f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:57:23.927700 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.927641 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12bcfb3a1cdb54b757405d90e15ad22f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal\" (UID: \"12bcfb3a1cdb54b757405d90e15ad22f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:57:23.927700 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:23.927660 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/14ca05fc22f96fd5090f819c755a4a44-config\") pod \"kube-apiserver-proxy-ip-10-0-128-129.us-east-2.compute.internal\" (UID: \"14ca05fc22f96fd5090f819c755a4a44\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:57:24.028469 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:24.028439 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-129.us-east-2.compute.internal\" not found" Nov 04 00:57:24.028557 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:24.028474 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/12bcfb3a1cdb54b757405d90e15ad22f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal\" (UID: \"12bcfb3a1cdb54b757405d90e15ad22f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:57:24.028557 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:24.028497 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12bcfb3a1cdb54b757405d90e15ad22f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal\" (UID: \"12bcfb3a1cdb54b757405d90e15ad22f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:57:24.028557 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:24.028513 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/14ca05fc22f96fd5090f819c755a4a44-config\") pod \"kube-apiserver-proxy-ip-10-0-128-129.us-east-2.compute.internal\" (UID: \"14ca05fc22f96fd5090f819c755a4a44\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:57:24.028681 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:24.028563 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/14ca05fc22f96fd5090f819c755a4a44-config\") pod \"kube-apiserver-proxy-ip-10-0-128-129.us-east-2.compute.internal\" (UID: \"14ca05fc22f96fd5090f819c755a4a44\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:57:24.028681 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:24.028582 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12bcfb3a1cdb54b757405d90e15ad22f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal\" (UID: \"12bcfb3a1cdb54b757405d90e15ad22f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:57:24.028681 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:24.028590 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/12bcfb3a1cdb54b757405d90e15ad22f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal\" (UID: \"12bcfb3a1cdb54b757405d90e15ad22f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:57:24.128853 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:24.128805 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-129.us-east-2.compute.internal\" not found" Nov 04 00:57:24.196034 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:24.196006 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:57:24.197092 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:24.197078 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:57:24.229711 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:24.229684 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-129.us-east-2.compute.internal\" not found" Nov 04 00:57:24.330083 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:24.330045 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-129.us-east-2.compute.internal\" not found" Nov 04 00:57:24.357225 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:24.357203 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 04 00:57:24.357604 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:24.357338 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 04 00:57:24.357604 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:24.357373 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 04 00:57:24.423802 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:24.423724 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-03 00:52:23 +0000 UTC" deadline="2028-08-12 07:52:02.804240374 +0000 UTC" Nov 04 00:57:24.423802 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:24.423762 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15534h54m38.380480439s" Nov 04 00:57:24.431066 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:24.431048 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-129.us-east-2.compute.internal\" not found" Nov 04 00:57:24.531478 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:24.531457 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-129.us-east-2.compute.internal\" not found" Nov 04 00:57:24.631808 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:24.631786 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-129.us-east-2.compute.internal\" not found" Nov 04 00:57:24.673894 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:24.673842 2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:57:24.725896 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:24.725872 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:57:24.736940 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:24.736914 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 00:57:24.739901 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:24.739887 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:57:24.747516 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:24.747504 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 00:57:24.976630 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:24.976461 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14ca05fc22f96fd5090f819c755a4a44.slice/crio-e158cae34761dcb7e8d465156deae870bc77549e90d3d942895cd9c8fa82ab4a WatchSource:0}: Error finding container e158cae34761dcb7e8d465156deae870bc77549e90d3d942895cd9c8fa82ab4a: Status 404 returned error can't find the container with id e158cae34761dcb7e8d465156deae870bc77549e90d3d942895cd9c8fa82ab4a Nov 04 00:57:24.976874 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:24.976852 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12bcfb3a1cdb54b757405d90e15ad22f.slice/crio-35a47295d9f86275eb10f7f578cc2d21fe2637b5c452766ffdbf53d592efa0d2 WatchSource:0}: Error finding container 35a47295d9f86275eb10f7f578cc2d21fe2637b5c452766ffdbf53d592efa0d2: Status 404 returned error can't find the container with id 35a47295d9f86275eb10f7f578cc2d21fe2637b5c452766ffdbf53d592efa0d2 Nov 04 00:57:24.983270 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:24.983253 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 00:57:24.995864 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:24.995834 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:57:25.265137 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.265083 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:57:25.400394 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.400373 2577 apiserver.go:52] "Watching apiserver" Nov 04 00:57:25.409658 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.409634 2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 04 00:57:25.409916 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.409900 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-2x7kl","kube-system/kube-apiserver-proxy-ip-10-0-128-129.us-east-2.compute.internal","openshift-network-diagnostics/network-check-target-289kb","openshift-network-operator/iptables-alerter-96mj9","kube-system/konnectivity-agent-nb955","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq","openshift-cluster-node-tuning-operator/tuned-q8w8s","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-xwbdv","openshift-dns/node-resolver-h4s87","openshift-image-registry/node-ca-z4snb","openshift-multus/multus-7z64n","openshift-multus/multus-additional-cni-plugins-jx882","openshift-multus/network-metrics-daemon-l4d28"] Nov 04 00:57:25.413799 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.413784 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.415845 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.415822 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:57:25.417756 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.417739 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:57:25.417858 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:25.417790 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289kb" podUID="9481376a-a63f-46e2-a605-fe4bb019c331" Nov 04 00:57:25.418436 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.418398 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:57:25.418671 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.418645 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 04 00:57:25.418773 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.418758 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ln4qv\"" Nov 04 00:57:25.419887 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.419802 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-96mj9" Nov 04 00:57:25.421127 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.421107 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 04 00:57:25.421295 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.421280 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 04 00:57:25.421516 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.421498 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 04 00:57:25.421586 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.421498 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 04 00:57:25.421586 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.421524 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 04 00:57:25.421586 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.421564 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sknxk\"" Nov 04 00:57:25.422143 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.422128 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nb955" Nov 04 00:57:25.424302 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.424283 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:57:25.424401 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.424358 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zgrbh\"" Nov 04 00:57:25.424883 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.424863 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 04 00:57:25.425032 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.424964 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 04 00:57:25.425032 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.425002 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:57:25.425032 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.425008 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 04 00:57:25.426686 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.426666 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.429181 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.429167 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:57:25.429240 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:25.429223 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2x7kl" podUID="bac70f0f-4585-42fe-abba-916ac41fcc62" Nov 04 00:57:25.431618 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.431599 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:57:25.431715 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:25.431690 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d28" podUID="5d232021-de4b-4240-8877-e99fe3563101" Nov 04 00:57:25.433762 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.433747 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h4s87" Nov 04 00:57:25.436707 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.436273 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z4snb" Nov 04 00:57:25.437106 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.436975 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:57:25.437199 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.437033 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-run-netns\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.437199 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.437175 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-run-systemd\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.437291 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.437205 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-log-socket\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.437291 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.437233 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/26470ab5-58d3-4b17-bb82-e1b20803e7db-ovn-node-metrics-cert\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.437291 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.437265 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d7sj\" (UniqueName: \"kubernetes.io/projected/9481376a-a63f-46e2-a605-fe4bb019c331-kube-api-access-6d7sj\") pod \"network-check-target-289kb\" (UID: \"9481376a-a63f-46e2-a605-fe4bb019c331\") " pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:57:25.437410 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.437296 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-systemd-units\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.437410 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.437325 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-systemd\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.437410 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.437357 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7kw6\" (UniqueName: \"kubernetes.io/projected/d8d094b5-dae6-4655-9425-c10f62f41f07-kube-api-access-b7kw6\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.437552 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.437408 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bbfd4978-1740-4bf1-b18b-f2d7665fa421-cnibin\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:57:25.437609 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.437554 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-cni-netd\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.437609 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.437586 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/26470ab5-58d3-4b17-bb82-e1b20803e7db-ovnkube-config\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.437717 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.437638 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x756c\" (UniqueName: \"kubernetes.io/projected/bbfd4978-1740-4bf1-b18b-f2d7665fa421-kube-api-access-x756c\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:57:25.437717 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.437687 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-sysctl-d\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.437819 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.437733 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-var-lib-kubelet\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.437819 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.437759 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-tuned\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.437913 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.437812 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbfd4978-1740-4bf1-b18b-f2d7665fa421-system-cni-dir\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:57:25.437913 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.437884 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bbfd4978-1740-4bf1-b18b-f2d7665fa421-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:57:25.438038 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.437942 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-device-dir\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:57:25.438038 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.437985 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-etc-selinux\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:57:25.438135 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.438048 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-var-lib-openvswitch\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.438135 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.438084 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-kubernetes\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.438230 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.438136 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bbfd4978-1740-4bf1-b18b-f2d7665fa421-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:57:25.438230 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.438189 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69mrb\" (UniqueName: \"kubernetes.io/projected/79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe-kube-api-access-69mrb\") pod \"iptables-alerter-96mj9\" (UID: \"79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe\") " pod="openshift-network-operator/iptables-alerter-96mj9" Nov 04 00:57:25.438320 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.438233 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-kubelet\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.438320 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.438288 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-etc-openvswitch\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.438409 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.438321 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.438409 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.438377 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-sysconfig\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.438495 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.438401 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/26470ab5-58d3-4b17-bb82-e1b20803e7db-env-overrides\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.438495 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.438435 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-run-ovn-kubernetes\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.438495 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.438468 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/26470ab5-58d3-4b17-bb82-e1b20803e7db-ovnkube-script-lib\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.438626 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.438537 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-modprobe-d\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.438626 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.438571 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-lib-modules\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.438626 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.438599 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe-iptables-alerter-script\") pod \"iptables-alerter-96mj9\" (UID: \"79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe\") " pod="openshift-network-operator/iptables-alerter-96mj9" Nov 04 00:57:25.438769 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.438632 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-run-openvswitch\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.438769 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.438664 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmn8n\" (UniqueName: \"kubernetes.io/projected/d2344442-689b-49f6-a97b-365500025b2d-kube-api-access-wmn8n\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:57:25.438769 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.438744 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-run-ovn\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.438901 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.438803 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-node-log\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.438901 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.438840 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-cni-bin\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.438901 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.438894 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpl2j\" (UniqueName: \"kubernetes.io/projected/26470ab5-58d3-4b17-bb82-e1b20803e7db-kube-api-access-rpl2j\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.439158 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.438953 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-run\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.439158 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.438984 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-sys\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.439158 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.439012 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-host\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.439158 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.439068 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bbfd4978-1740-4bf1-b18b-f2d7665fa421-cni-binary-copy\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:57:25.439158 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.439100 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bbfd4978-1740-4bf1-b18b-f2d7665fa421-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:57:25.439158 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.439130 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-socket-dir\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:57:25.439158 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.439152 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.440661 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.439159 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/de6f6762-66c6-42c0-bf0d-7966483a040d-konnectivity-ca\") pod \"konnectivity-agent-nb955\" (UID: \"de6f6762-66c6-42c0-bf0d-7966483a040d\") " pod="kube-system/konnectivity-agent-nb955" Nov 04 00:57:25.440661 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.439192 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8d094b5-dae6-4655-9425-c10f62f41f07-tmp\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.440661 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.439330 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-sysctl-conf\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.440661 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.439377 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-sys-fs\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:57:25.440661 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.439427 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-registration-dir\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:57:25.440661 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.439463 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe-host-slash\") pod \"iptables-alerter-96mj9\" (UID: \"79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe\") " pod="openshift-network-operator/iptables-alerter-96mj9" Nov 04 00:57:25.440661 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.439487 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/de6f6762-66c6-42c0-bf0d-7966483a040d-agent-certs\") pod \"konnectivity-agent-nb955\" (UID: \"de6f6762-66c6-42c0-bf0d-7966483a040d\") " pod="kube-system/konnectivity-agent-nb955" Nov 04 00:57:25.440661 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.439523 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-slash\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.451905 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.451889 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 04 00:57:25.473882 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.473864 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-k79c7" Nov 04 00:57:25.486765 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.486741 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-k79c7" Nov 04 00:57:25.526593 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.526553 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 04 00:57:25.531268 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.531253 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 04 00:57:25.531457 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.531436 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 04 00:57:25.531457 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.531440 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 04 00:57:25.531572 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.531537 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 04 00:57:25.532474 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.532457 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 04 00:57:25.532586 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.532524 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-psr9r\"" Nov 04 00:57:25.532821 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.532809 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 04 00:57:25.533010 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.532984 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 04 00:57:25.533099 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.533029 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 04 00:57:25.533763 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.533748 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 04 00:57:25.533824 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.533807 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 04 00:57:25.533982 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.533957 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 04 00:57:25.533982 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.533959 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 04 00:57:25.534096 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.533981 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 04 00:57:25.534096 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.533993 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kqwbk\"" Nov 04 00:57:25.534096 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.533996 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 04 00:57:25.534562 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.534537 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zx855\"" Nov 04 00:57:25.534562 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.534556 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 04 00:57:25.534677 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.534584 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 04 00:57:25.534677 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.534639 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9jwd5\"" Nov 04 00:57:25.535363 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.535350 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4k8lm\"" Nov 04 00:57:25.535465 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.535454 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 04 00:57:25.535495 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.535486 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-xdt54\"" Nov 04 00:57:25.540550 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.540534 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-kubernetes\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.540594 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.540560 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bbfd4978-1740-4bf1-b18b-f2d7665fa421-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:57:25.540594 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.540577 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69mrb\" (UniqueName: \"kubernetes.io/projected/79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe-kube-api-access-69mrb\") pod \"iptables-alerter-96mj9\" (UID: \"79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe\") " pod="openshift-network-operator/iptables-alerter-96mj9" Nov 04 00:57:25.540648 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.540598 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-kubelet\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.540676 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.540643 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-kubernetes\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.540676 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.540643 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-kubelet\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.540730 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.540679 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-etc-openvswitch\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.540730 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.540699 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-run-k8s-cni-cncf-io\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.540730 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.540713 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-run-multus-certs\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.540845 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.540733 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-etc-kubernetes\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.540845 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.540713 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bbfd4978-1740-4bf1-b18b-f2d7665fa421-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:57:25.540845 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.540743 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-etc-openvswitch\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.540845 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.540791 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.540974 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.540853 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-run-netns\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.540974 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.540856 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.540974 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.540874 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld5mg\" (UniqueName: \"kubernetes.io/projected/cdcb63f9-d455-4105-8911-33dc1d3dedad-kube-api-access-ld5mg\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.540974 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.540904 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-sysconfig\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.540974 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.540966 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-sysconfig\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.541101 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.540988 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/26470ab5-58d3-4b17-bb82-e1b20803e7db-env-overrides\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.541101 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541013 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-system-cni-dir\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.541101 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541031 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-run-ovn-kubernetes\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.541101 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541048 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cdcb63f9-d455-4105-8911-33dc1d3dedad-cni-binary-copy\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.541101 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541071 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-run-ovn-kubernetes\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.541101 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541087 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-multus-conf-dir\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.541327 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541107 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/26470ab5-58d3-4b17-bb82-e1b20803e7db-ovnkube-script-lib\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.541327 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541141 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxb5v\" (UniqueName: \"kubernetes.io/projected/5d232021-de4b-4240-8877-e99fe3563101-kube-api-access-qxb5v\") pod \"network-metrics-daemon-l4d28\" (UID: \"5d232021-de4b-4240-8877-e99fe3563101\") " pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:57:25.541327 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541187 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-cnibin\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.541327 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541217 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-modprobe-d\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.541327 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541240 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-lib-modules\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.541327 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541269 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe-iptables-alerter-script\") pod \"iptables-alerter-96mj9\" (UID: \"79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe\") " pod="openshift-network-operator/iptables-alerter-96mj9" Nov 04 00:57:25.541327 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541324 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-run-openvswitch\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.541563 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541349 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bac70f0f-4585-42fe-abba-916ac41fcc62-original-pull-secret\") pod \"global-pull-secret-syncer-2x7kl\" (UID: \"bac70f0f-4585-42fe-abba-916ac41fcc62\") " pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:57:25.541563 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541364 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-modprobe-d\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.541563 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541374 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d232021-de4b-4240-8877-e99fe3563101-metrics-certs\") pod \"network-metrics-daemon-l4d28\" (UID: \"5d232021-de4b-4240-8877-e99fe3563101\") " pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:57:25.541563 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541406 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-multus-cni-dir\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.541563 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541416 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-run-openvswitch\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.541563 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541417 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/26470ab5-58d3-4b17-bb82-e1b20803e7db-env-overrides\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.541563 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541444 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-lib-modules\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.541563 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541451 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-var-lib-cni-bin\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.541563 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541490 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmn8n\" (UniqueName: \"kubernetes.io/projected/d2344442-689b-49f6-a97b-365500025b2d-kube-api-access-wmn8n\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:57:25.541563 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541513 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-run-ovn\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.541563 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541526 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-node-log\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.541563 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541557 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/26470ab5-58d3-4b17-bb82-e1b20803e7db-ovnkube-script-lib\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.542019 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541575 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-cni-bin\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.542019 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541576 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-run-ovn\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.542019 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541660 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-cni-bin\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.542019 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541672 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-node-log\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.542019 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541717 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpl2j\" (UniqueName: \"kubernetes.io/projected/26470ab5-58d3-4b17-bb82-e1b20803e7db-kube-api-access-rpl2j\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.542019 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541726 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe-iptables-alerter-script\") pod \"iptables-alerter-96mj9\" (UID: \"79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe\") " pod="openshift-network-operator/iptables-alerter-96mj9" Nov 04 00:57:25.542019 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541767 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cdcb63f9-d455-4105-8911-33dc1d3dedad-multus-daemon-config\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.542019 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541787 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-run\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.542019 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541801 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-sys\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.542019 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541840 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-host\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.542019 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541888 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-host\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.542019 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541907 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bbfd4978-1740-4bf1-b18b-f2d7665fa421-cni-binary-copy\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:57:25.542019 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541912 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-sys\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.542019 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.541977 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bbfd4978-1740-4bf1-b18b-f2d7665fa421-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:57:25.542019 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.542006 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-run\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.542674 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.542036 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-socket-dir\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:57:25.542674 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.542082 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/de6f6762-66c6-42c0-bf0d-7966483a040d-konnectivity-ca\") pod \"konnectivity-agent-nb955\" (UID: \"de6f6762-66c6-42c0-bf0d-7966483a040d\") " pod="kube-system/konnectivity-agent-nb955" Nov 04 00:57:25.542674 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.542105 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfzkw\" (UniqueName: \"kubernetes.io/projected/388103ca-b2f5-42b2-96ce-737f1ef00dc1-kube-api-access-sfzkw\") pod \"node-resolver-h4s87\" (UID: \"388103ca-b2f5-42b2-96ce-737f1ef00dc1\") " pod="openshift-dns/node-resolver-h4s87" Nov 04 00:57:25.542674 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.542127 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8d094b5-dae6-4655-9425-c10f62f41f07-tmp\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.542674 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.542160 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/61e0c4ee-edf6-4938-96c0-c79812e00840-serviceca\") pod \"node-ca-z4snb\" (UID: \"61e0c4ee-edf6-4938-96c0-c79812e00840\") " pod="openshift-image-registry/node-ca-z4snb" Nov 04 00:57:25.542674 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.542164 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-socket-dir\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:57:25.542674 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.542199 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-sysctl-conf\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.542674 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.542237 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-sys-fs\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:57:25.542674 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.542260 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb9nj\" (UniqueName: \"kubernetes.io/projected/61e0c4ee-edf6-4938-96c0-c79812e00840-kube-api-access-wb9nj\") pod \"node-ca-z4snb\" (UID: \"61e0c4ee-edf6-4938-96c0-c79812e00840\") " pod="openshift-image-registry/node-ca-z4snb" Nov 04 00:57:25.542674 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.542282 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-registration-dir\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:57:25.542674 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.542302 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe-host-slash\") pod \"iptables-alerter-96mj9\" (UID: \"79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe\") " pod="openshift-network-operator/iptables-alerter-96mj9" Nov 04 00:57:25.542674 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.542324 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/de6f6762-66c6-42c0-bf0d-7966483a040d-agent-certs\") pod \"konnectivity-agent-nb955\" (UID: \"de6f6762-66c6-42c0-bf0d-7966483a040d\") " pod="kube-system/konnectivity-agent-nb955" Nov 04 00:57:25.542674 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.542346 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-slash\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.542674 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.542368 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-sys-fs\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:57:25.542674 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.542370 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-var-lib-cni-multus\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.542674 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.542370 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-registration-dir\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:57:25.542674 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.542325 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-sysctl-conf\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.543406 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.542409 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bbfd4978-1740-4bf1-b18b-f2d7665fa421-cni-binary-copy\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:57:25.543406 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.542417 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-slash\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.543406 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.542449 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe-host-slash\") pod \"iptables-alerter-96mj9\" (UID: \"79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe\") " pod="openshift-network-operator/iptables-alerter-96mj9" Nov 04 00:57:25.543406 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.542454 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bbfd4978-1740-4bf1-b18b-f2d7665fa421-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:57:25.543406 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.542451 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:57:25.543406 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.542455 2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 04 00:57:25.543406 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.542558 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:57:25.543406 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.542636 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-run-netns\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.543406 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.542506 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-run-netns\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.543406 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.542990 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/de6f6762-66c6-42c0-bf0d-7966483a040d-konnectivity-ca\") pod \"konnectivity-agent-nb955\" (UID: \"de6f6762-66c6-42c0-bf0d-7966483a040d\") " pod="kube-system/konnectivity-agent-nb955" Nov 04 00:57:25.543406 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.543001 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-run-systemd\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.543406 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.543024 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-log-socket\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.543406 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.543045 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/26470ab5-58d3-4b17-bb82-e1b20803e7db-ovn-node-metrics-cert\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.543406 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.543069 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-multus-socket-dir-parent\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.543406 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.543076 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-log-socket\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.543406 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.543046 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-run-systemd\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.543406 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.543093 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-hostroot\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.543406 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.543118 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d7sj\" (UniqueName: \"kubernetes.io/projected/9481376a-a63f-46e2-a605-fe4bb019c331-kube-api-access-6d7sj\") pod \"network-check-target-289kb\" (UID: \"9481376a-a63f-46e2-a605-fe4bb019c331\") " pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:57:25.543874 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.543139 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-systemd-units\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.543874 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.543161 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/388103ca-b2f5-42b2-96ce-737f1ef00dc1-tmp-dir\") pod \"node-resolver-h4s87\" (UID: \"388103ca-b2f5-42b2-96ce-737f1ef00dc1\") " pod="openshift-dns/node-resolver-h4s87" Nov 04 00:57:25.543874 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.543183 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-var-lib-kubelet\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.543874 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.543206 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-systemd\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.543874 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.543227 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7kw6\" (UniqueName: \"kubernetes.io/projected/d8d094b5-dae6-4655-9425-c10f62f41f07-kube-api-access-b7kw6\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.543874 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.543248 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bbfd4978-1740-4bf1-b18b-f2d7665fa421-cnibin\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:57:25.543874 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.543270 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/388103ca-b2f5-42b2-96ce-737f1ef00dc1-hosts-file\") pod \"node-resolver-h4s87\" (UID: \"388103ca-b2f5-42b2-96ce-737f1ef00dc1\") " pod="openshift-dns/node-resolver-h4s87" Nov 04 00:57:25.543874 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.543297 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-cni-netd\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.543874 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.543321 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/26470ab5-58d3-4b17-bb82-e1b20803e7db-ovnkube-config\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.543874 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.543358 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x756c\" (UniqueName: \"kubernetes.io/projected/bbfd4978-1740-4bf1-b18b-f2d7665fa421-kube-api-access-x756c\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:57:25.543874 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.543499 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-systemd-units\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.543874 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.543632 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-systemd\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.543874 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.543826 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bbfd4978-1740-4bf1-b18b-f2d7665fa421-cnibin\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:57:25.544352 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.544059 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-cni-netd\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.544352 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.544064 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-sysctl-d\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.544352 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.544111 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-var-lib-kubelet\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.544352 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.544158 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-tuned\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.544352 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.544197 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbfd4978-1740-4bf1-b18b-f2d7665fa421-system-cni-dir\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:57:25.544352 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.544204 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-var-lib-kubelet\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.544352 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.544159 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-sysctl-d\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.544352 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.544241 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bbfd4978-1740-4bf1-b18b-f2d7665fa421-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:57:25.544352 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.544267 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-device-dir\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:57:25.544352 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.544306 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-etc-selinux\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:57:25.544352 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.544315 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbfd4978-1740-4bf1-b18b-f2d7665fa421-system-cni-dir\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:57:25.544352 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.544335 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-var-lib-openvswitch\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.544352 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.544335 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-device-dir\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:57:25.544969 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.544383 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-var-lib-openvswitch\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.544969 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.544412 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bac70f0f-4585-42fe-abba-916ac41fcc62-kubelet-config\") pod \"global-pull-secret-syncer-2x7kl\" (UID: \"bac70f0f-4585-42fe-abba-916ac41fcc62\") " pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:57:25.544969 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.544435 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bac70f0f-4585-42fe-abba-916ac41fcc62-dbus\") pod \"global-pull-secret-syncer-2x7kl\" (UID: \"bac70f0f-4585-42fe-abba-916ac41fcc62\") " pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:57:25.544969 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.544436 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-etc-selinux\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:57:25.544969 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.544456 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61e0c4ee-edf6-4938-96c0-c79812e00840-host\") pod \"node-ca-z4snb\" (UID: \"61e0c4ee-edf6-4938-96c0-c79812e00840\") " pod="openshift-image-registry/node-ca-z4snb" Nov 04 00:57:25.544969 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.544475 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/26470ab5-58d3-4b17-bb82-e1b20803e7db-ovnkube-config\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.545148 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.544979 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bbfd4978-1740-4bf1-b18b-f2d7665fa421-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:57:25.545777 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.545753 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/26470ab5-58d3-4b17-bb82-e1b20803e7db-ovn-node-metrics-cert\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.546144 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.546128 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8d094b5-dae6-4655-9425-c10f62f41f07-tmp\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.546324 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.546309 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/de6f6762-66c6-42c0-bf0d-7966483a040d-agent-certs\") pod \"konnectivity-agent-nb955\" (UID: \"de6f6762-66c6-42c0-bf0d-7966483a040d\") " pod="kube-system/konnectivity-agent-nb955" Nov 04 00:57:25.546368 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.546314 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-tuned\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.562281 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:25.562251 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 00:57:25.562431 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:25.562375 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 00:57:25.562476 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:25.562436 2577 projected.go:196] Error preparing data for projected volume kube-api-access-6d7sj for pod openshift-network-diagnostics/network-check-target-289kb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:57:25.562504 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:25.562490 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9481376a-a63f-46e2-a605-fe4bb019c331-kube-api-access-6d7sj podName:9481376a-a63f-46e2-a605-fe4bb019c331 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:26.062471706 +0000 UTC m=+2.974638217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6d7sj" (UniqueName: "kubernetes.io/projected/9481376a-a63f-46e2-a605-fe4bb019c331-kube-api-access-6d7sj") pod "network-check-target-289kb" (UID: "9481376a-a63f-46e2-a605-fe4bb019c331") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:57:25.562723 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.562705 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69mrb\" (UniqueName: \"kubernetes.io/projected/79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe-kube-api-access-69mrb\") pod \"iptables-alerter-96mj9\" (UID: \"79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe\") " pod="openshift-network-operator/iptables-alerter-96mj9" Nov 04 00:57:25.562881 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.562865 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmn8n\" (UniqueName: \"kubernetes.io/projected/d2344442-689b-49f6-a97b-365500025b2d-kube-api-access-wmn8n\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:57:25.562952 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.562888 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x756c\" (UniqueName: \"kubernetes.io/projected/bbfd4978-1740-4bf1-b18b-f2d7665fa421-kube-api-access-x756c\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:57:25.563034 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.563016 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpl2j\" (UniqueName: \"kubernetes.io/projected/26470ab5-58d3-4b17-bb82-e1b20803e7db-kube-api-access-rpl2j\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.563815 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.563801 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7kw6\" (UniqueName: \"kubernetes.io/projected/d8d094b5-dae6-4655-9425-c10f62f41f07-kube-api-access-b7kw6\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.645420 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645393 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/388103ca-b2f5-42b2-96ce-737f1ef00dc1-tmp-dir\") pod \"node-resolver-h4s87\" (UID: \"388103ca-b2f5-42b2-96ce-737f1ef00dc1\") " pod="openshift-dns/node-resolver-h4s87" Nov 04 00:57:25.645420 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645421 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-var-lib-kubelet\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.645550 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645437 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/388103ca-b2f5-42b2-96ce-737f1ef00dc1-hosts-file\") pod \"node-resolver-h4s87\" (UID: \"388103ca-b2f5-42b2-96ce-737f1ef00dc1\") " pod="openshift-dns/node-resolver-h4s87" Nov 04 00:57:25.645550 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645453 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bac70f0f-4585-42fe-abba-916ac41fcc62-kubelet-config\") pod \"global-pull-secret-syncer-2x7kl\" (UID: \"bac70f0f-4585-42fe-abba-916ac41fcc62\") " pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:57:25.645550 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645466 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bac70f0f-4585-42fe-abba-916ac41fcc62-dbus\") pod \"global-pull-secret-syncer-2x7kl\" (UID: \"bac70f0f-4585-42fe-abba-916ac41fcc62\") " pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:57:25.645550 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645478 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61e0c4ee-edf6-4938-96c0-c79812e00840-host\") pod \"node-ca-z4snb\" (UID: \"61e0c4ee-edf6-4938-96c0-c79812e00840\") " pod="openshift-image-registry/node-ca-z4snb" Nov 04 00:57:25.645550 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645491 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-run-k8s-cni-cncf-io\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.645550 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645494 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-var-lib-kubelet\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.645550 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645503 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-run-multus-certs\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.645550 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645533 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-run-multus-certs\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.645550 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645548 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-etc-kubernetes\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.645897 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645572 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-run-netns\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.645897 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645577 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-run-k8s-cni-cncf-io\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.645897 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645595 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld5mg\" (UniqueName: \"kubernetes.io/projected/cdcb63f9-d455-4105-8911-33dc1d3dedad-kube-api-access-ld5mg\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.645897 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645600 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-etc-kubernetes\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.645897 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645608 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61e0c4ee-edf6-4938-96c0-c79812e00840-host\") pod \"node-ca-z4snb\" (UID: \"61e0c4ee-edf6-4938-96c0-c79812e00840\") " pod="openshift-image-registry/node-ca-z4snb" Nov 04 00:57:25.645897 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645571 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/388103ca-b2f5-42b2-96ce-737f1ef00dc1-hosts-file\") pod \"node-resolver-h4s87\" (UID: \"388103ca-b2f5-42b2-96ce-737f1ef00dc1\") " pod="openshift-dns/node-resolver-h4s87" Nov 04 00:57:25.645897 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645612 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bac70f0f-4585-42fe-abba-916ac41fcc62-dbus\") pod \"global-pull-secret-syncer-2x7kl\" (UID: \"bac70f0f-4585-42fe-abba-916ac41fcc62\") " pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:57:25.645897 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645613 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bac70f0f-4585-42fe-abba-916ac41fcc62-kubelet-config\") pod \"global-pull-secret-syncer-2x7kl\" (UID: \"bac70f0f-4585-42fe-abba-916ac41fcc62\") " pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:57:25.645897 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645619 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-system-cni-dir\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.645897 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645644 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-run-netns\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.645897 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645665 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-system-cni-dir\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.645897 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645673 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cdcb63f9-d455-4105-8911-33dc1d3dedad-cni-binary-copy\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.645897 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645727 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-multus-conf-dir\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.645897 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645794 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-multus-conf-dir\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.645897 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645792 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/388103ca-b2f5-42b2-96ce-737f1ef00dc1-tmp-dir\") pod \"node-resolver-h4s87\" (UID: \"388103ca-b2f5-42b2-96ce-737f1ef00dc1\") " pod="openshift-dns/node-resolver-h4s87" Nov 04 00:57:25.645897 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645798 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxb5v\" (UniqueName: \"kubernetes.io/projected/5d232021-de4b-4240-8877-e99fe3563101-kube-api-access-qxb5v\") pod \"network-metrics-daemon-l4d28\" (UID: \"5d232021-de4b-4240-8877-e99fe3563101\") " pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:57:25.645897 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645835 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-cnibin\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.645897 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645861 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bac70f0f-4585-42fe-abba-916ac41fcc62-original-pull-secret\") pod \"global-pull-secret-syncer-2x7kl\" (UID: \"bac70f0f-4585-42fe-abba-916ac41fcc62\") " pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:57:25.646604 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645883 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d232021-de4b-4240-8877-e99fe3563101-metrics-certs\") pod \"network-metrics-daemon-l4d28\" (UID: \"5d232021-de4b-4240-8877-e99fe3563101\") " pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:57:25.646604 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645890 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-cnibin\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.646604 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645912 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-multus-cni-dir\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.646604 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645950 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-var-lib-cni-bin\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.646604 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:25.645964 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 00:57:25.646604 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.645994 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-var-lib-cni-bin\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.646604 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:25.646007 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bac70f0f-4585-42fe-abba-916ac41fcc62-original-pull-secret podName:bac70f0f-4585-42fe-abba-916ac41fcc62 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:26.1459965 +0000 UTC m=+3.058163008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bac70f0f-4585-42fe-abba-916ac41fcc62-original-pull-secret") pod "global-pull-secret-syncer-2x7kl" (UID: "bac70f0f-4585-42fe-abba-916ac41fcc62") : object "kube-system"/"original-pull-secret" not registered Nov 04 00:57:25.646604 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:25.646015 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:57:25.646604 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.646040 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-multus-cni-dir\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.646604 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.646050 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cdcb63f9-d455-4105-8911-33dc1d3dedad-multus-daemon-config\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.646604 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:25.646066 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d232021-de4b-4240-8877-e99fe3563101-metrics-certs podName:5d232021-de4b-4240-8877-e99fe3563101 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:26.146049496 +0000 UTC m=+3.058215994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d232021-de4b-4240-8877-e99fe3563101-metrics-certs") pod "network-metrics-daemon-l4d28" (UID: "5d232021-de4b-4240-8877-e99fe3563101") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:57:25.646604 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.646105 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfzkw\" (UniqueName: \"kubernetes.io/projected/388103ca-b2f5-42b2-96ce-737f1ef00dc1-kube-api-access-sfzkw\") pod \"node-resolver-h4s87\" (UID: \"388103ca-b2f5-42b2-96ce-737f1ef00dc1\") " pod="openshift-dns/node-resolver-h4s87" Nov 04 00:57:25.646604 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.646130 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/61e0c4ee-edf6-4938-96c0-c79812e00840-serviceca\") pod \"node-ca-z4snb\" (UID: \"61e0c4ee-edf6-4938-96c0-c79812e00840\") " pod="openshift-image-registry/node-ca-z4snb" Nov 04 00:57:25.646604 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.646156 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb9nj\" (UniqueName: \"kubernetes.io/projected/61e0c4ee-edf6-4938-96c0-c79812e00840-kube-api-access-wb9nj\") pod \"node-ca-z4snb\" (UID: \"61e0c4ee-edf6-4938-96c0-c79812e00840\") " pod="openshift-image-registry/node-ca-z4snb" Nov 04 00:57:25.646604 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.646165 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cdcb63f9-d455-4105-8911-33dc1d3dedad-cni-binary-copy\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.646604 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.646182 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-var-lib-cni-multus\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.646604 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.646207 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-multus-socket-dir-parent\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.647182 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.646223 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-var-lib-cni-multus\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.647182 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.646227 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-hostroot\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.647182 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.646260 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-hostroot\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.647182 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.646289 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-multus-socket-dir-parent\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.647182 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.646452 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cdcb63f9-d455-4105-8911-33dc1d3dedad-multus-daemon-config\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.647182 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.646529 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/61e0c4ee-edf6-4938-96c0-c79812e00840-serviceca\") pod \"node-ca-z4snb\" (UID: \"61e0c4ee-edf6-4938-96c0-c79812e00840\") " pod="openshift-image-registry/node-ca-z4snb" Nov 04 00:57:25.662255 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.662239 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld5mg\" (UniqueName: \"kubernetes.io/projected/cdcb63f9-d455-4105-8911-33dc1d3dedad-kube-api-access-ld5mg\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.664742 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.664724 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxb5v\" (UniqueName: \"kubernetes.io/projected/5d232021-de4b-4240-8877-e99fe3563101-kube-api-access-qxb5v\") pod \"network-metrics-daemon-l4d28\" (UID: \"5d232021-de4b-4240-8877-e99fe3563101\") " pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:57:25.666413 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.666397 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfzkw\" (UniqueName: \"kubernetes.io/projected/388103ca-b2f5-42b2-96ce-737f1ef00dc1-kube-api-access-sfzkw\") pod \"node-resolver-h4s87\" (UID: \"388103ca-b2f5-42b2-96ce-737f1ef00dc1\") " pod="openshift-dns/node-resolver-h4s87" Nov 04 00:57:25.666735 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.666714 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb9nj\" (UniqueName: \"kubernetes.io/projected/61e0c4ee-edf6-4938-96c0-c79812e00840-kube-api-access-wb9nj\") pod \"node-ca-z4snb\" (UID: \"61e0c4ee-edf6-4938-96c0-c79812e00840\") " pod="openshift-image-registry/node-ca-z4snb" Nov 04 00:57:25.727621 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.727461 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:57:25.736524 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:25.736501 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8d094b5_dae6_4655_9425_c10f62f41f07.slice/crio-fa6388cb1fe29a1039b00eacc3ead60ba6d1193114bde5a9ad9d23a06db5360d WatchSource:0}: Error finding container fa6388cb1fe29a1039b00eacc3ead60ba6d1193114bde5a9ad9d23a06db5360d: Status 404 returned error can't find the container with id fa6388cb1fe29a1039b00eacc3ead60ba6d1193114bde5a9ad9d23a06db5360d Nov 04 00:57:25.747647 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.747627 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-96mj9" Nov 04 00:57:25.751292 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.751269 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nb955" Nov 04 00:57:25.752428 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.752297 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:57:25.753684 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.753647 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z4snb" Nov 04 00:57:25.756565 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:25.756545 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b6b73a_fb4d_4a89_8d7e_efbb3c463bfe.slice/crio-d164a428de2889bd890be21449cde06ccd895263d6ab8392bc534326d6d30224 WatchSource:0}: Error finding container d164a428de2889bd890be21449cde06ccd895263d6ab8392bc534326d6d30224: Status 404 returned error can't find the container with id d164a428de2889bd890be21449cde06ccd895263d6ab8392bc534326d6d30224 Nov 04 00:57:25.760889 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.760830 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:57:25.770356 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.769956 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" event={"ID":"d8d094b5-dae6-4655-9425-c10f62f41f07","Type":"ContainerStarted","Data":"fa6388cb1fe29a1039b00eacc3ead60ba6d1193114bde5a9ad9d23a06db5360d"} Nov 04 00:57:25.771724 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.771702 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal" event={"ID":"12bcfb3a1cdb54b757405d90e15ad22f","Type":"ContainerStarted","Data":"35a47295d9f86275eb10f7f578cc2d21fe2637b5c452766ffdbf53d592efa0d2"} Nov 04 00:57:25.772339 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.772320 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h4s87" Nov 04 00:57:25.773052 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:25.773030 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2344442_689b_49f6_a97b_365500025b2d.slice/crio-c5ac65ae8ef366a33b346f3b55376013920838985c353f9b7f9b702d5ab7d0ca WatchSource:0}: Error finding container c5ac65ae8ef366a33b346f3b55376013920838985c353f9b7f9b702d5ab7d0ca: Status 404 returned error can't find the container with id c5ac65ae8ef366a33b346f3b55376013920838985c353f9b7f9b702d5ab7d0ca Nov 04 00:57:25.773144 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.773111 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-129.us-east-2.compute.internal" event={"ID":"14ca05fc22f96fd5090f819c755a4a44","Type":"ContainerStarted","Data":"e158cae34761dcb7e8d465156deae870bc77549e90d3d942895cd9c8fa82ab4a"} Nov 04 00:57:25.774286 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.774270 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jx882" event={"ID":"bbfd4978-1740-4bf1-b18b-f2d7665fa421","Type":"ContainerStarted","Data":"75a6c1a77d683dc4e2c1c913eb203d9aba20647f8a731e4f4baa0f3a63ebaea6"} Nov 04 00:57:25.774844 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.774825 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7z64n" Nov 04 00:57:25.775491 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.775428 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nb955" event={"ID":"de6f6762-66c6-42c0-bf0d-7966483a040d","Type":"ContainerStarted","Data":"2f0063099a047671847bd4c92020cb142ff4d070dffaf850669d69aef2b05048"} Nov 04 00:57:25.776917 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.776861 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-96mj9" event={"ID":"79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe","Type":"ContainerStarted","Data":"d164a428de2889bd890be21449cde06ccd895263d6ab8392bc534326d6d30224"} Nov 04 00:57:25.781286 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.781269 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:25.782271 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:25.782239 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod388103ca_b2f5_42b2_96ce_737f1ef00dc1.slice/crio-30645fbd813ee7e2ef928a485967a08410099870bd752dd7c391d0b10d67ad87 WatchSource:0}: Error finding container 30645fbd813ee7e2ef928a485967a08410099870bd752dd7c391d0b10d67ad87: Status 404 returned error can't find the container with id 30645fbd813ee7e2ef928a485967a08410099870bd752dd7c391d0b10d67ad87 Nov 04 00:57:25.785822 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:25.785619 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdcb63f9_d455_4105_8911_33dc1d3dedad.slice/crio-4cfc28c7646e8a126052a353231f0aab20b4bfe1e31e6b7488dc447085ba219a WatchSource:0}: Error finding container 4cfc28c7646e8a126052a353231f0aab20b4bfe1e31e6b7488dc447085ba219a: Status 404 returned error can't find the container with id 4cfc28c7646e8a126052a353231f0aab20b4bfe1e31e6b7488dc447085ba219a Nov 04 00:57:25.797095 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:57:25.797076 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26470ab5_58d3_4b17_bb82_e1b20803e7db.slice/crio-478ad09b369a31d89d2db36a7d36a5cf1414649e3d3f3d53ba3e978d52b8859d WatchSource:0}: Error finding container 478ad09b369a31d89d2db36a7d36a5cf1414649e3d3f3d53ba3e978d52b8859d: Status 404 returned error can't find the container with id 478ad09b369a31d89d2db36a7d36a5cf1414649e3d3f3d53ba3e978d52b8859d Nov 04 00:57:25.851430 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:25.851404 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:57:26.149173 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:26.149106 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d7sj\" (UniqueName: \"kubernetes.io/projected/9481376a-a63f-46e2-a605-fe4bb019c331-kube-api-access-6d7sj\") pod \"network-check-target-289kb\" (UID: \"9481376a-a63f-46e2-a605-fe4bb019c331\") " pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:57:26.149173 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:26.149146 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bac70f0f-4585-42fe-abba-916ac41fcc62-original-pull-secret\") pod \"global-pull-secret-syncer-2x7kl\" (UID: \"bac70f0f-4585-42fe-abba-916ac41fcc62\") " pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:57:26.149173 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:26.149168 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d232021-de4b-4240-8877-e99fe3563101-metrics-certs\") pod \"network-metrics-daemon-l4d28\" (UID: \"5d232021-de4b-4240-8877-e99fe3563101\") " pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:57:26.149367 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:26.149236 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 00:57:26.149367 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:26.149292 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bac70f0f-4585-42fe-abba-916ac41fcc62-original-pull-secret podName:bac70f0f-4585-42fe-abba-916ac41fcc62 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:27.149279462 +0000 UTC m=+4.061445955 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bac70f0f-4585-42fe-abba-916ac41fcc62-original-pull-secret") pod "global-pull-secret-syncer-2x7kl" (UID: "bac70f0f-4585-42fe-abba-916ac41fcc62") : object "kube-system"/"original-pull-secret" not registered Nov 04 00:57:26.149606 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:26.149594 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:57:26.149606 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:26.149600 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 00:57:26.149666 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:26.149617 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 00:57:26.149666 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:26.149626 2577 projected.go:196] Error preparing data for projected volume kube-api-access-6d7sj for pod openshift-network-diagnostics/network-check-target-289kb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:57:26.149666 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:26.149636 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d232021-de4b-4240-8877-e99fe3563101-metrics-certs podName:5d232021-de4b-4240-8877-e99fe3563101 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:27.1496263 +0000 UTC m=+4.061792805 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d232021-de4b-4240-8877-e99fe3563101-metrics-certs") pod "network-metrics-daemon-l4d28" (UID: "5d232021-de4b-4240-8877-e99fe3563101") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:57:26.149666 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:26.149660 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9481376a-a63f-46e2-a605-fe4bb019c331-kube-api-access-6d7sj podName:9481376a-a63f-46e2-a605-fe4bb019c331 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:27.149648694 +0000 UTC m=+4.061815187 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6d7sj" (UniqueName: "kubernetes.io/projected/9481376a-a63f-46e2-a605-fe4bb019c331-kube-api-access-6d7sj") pod "network-check-target-289kb" (UID: "9481376a-a63f-46e2-a605-fe4bb019c331") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:57:26.490309 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:26.490153 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 00:52:25 +0000 UTC" deadline="2028-08-14 05:35:30.913815162 +0000 UTC" Nov 04 00:57:26.490309 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:26.490222 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15580h38m4.4235966s" Nov 04 00:57:26.765555 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:26.765484 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:57:26.765693 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:26.765614 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d28" podUID="5d232021-de4b-4240-8877-e99fe3563101" Nov 04 00:57:26.766040 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:26.766018 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:57:26.766119 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:26.766099 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289kb" podUID="9481376a-a63f-46e2-a605-fe4bb019c331" Nov 04 00:57:26.799455 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:26.799423 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" event={"ID":"26470ab5-58d3-4b17-bb82-e1b20803e7db","Type":"ContainerStarted","Data":"478ad09b369a31d89d2db36a7d36a5cf1414649e3d3f3d53ba3e978d52b8859d"} Nov 04 00:57:26.815671 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:26.815606 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z4snb" event={"ID":"61e0c4ee-edf6-4938-96c0-c79812e00840","Type":"ContainerStarted","Data":"949e8c265ebd46b758189e30223340b23e1fc52d75d3989e42ba951d4b663b6f"} Nov 04 00:57:26.819999 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:26.819873 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7z64n" event={"ID":"cdcb63f9-d455-4105-8911-33dc1d3dedad","Type":"ContainerStarted","Data":"4cfc28c7646e8a126052a353231f0aab20b4bfe1e31e6b7488dc447085ba219a"} Nov 04 00:57:26.834085 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:26.834053 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h4s87" event={"ID":"388103ca-b2f5-42b2-96ce-737f1ef00dc1","Type":"ContainerStarted","Data":"30645fbd813ee7e2ef928a485967a08410099870bd752dd7c391d0b10d67ad87"} Nov 04 00:57:26.836678 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:26.836654 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" event={"ID":"d2344442-689b-49f6-a97b-365500025b2d","Type":"ContainerStarted","Data":"c5ac65ae8ef366a33b346f3b55376013920838985c353f9b7f9b702d5ab7d0ca"} Nov 04 00:57:27.161174 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:27.160099 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bac70f0f-4585-42fe-abba-916ac41fcc62-original-pull-secret\") pod \"global-pull-secret-syncer-2x7kl\" (UID: \"bac70f0f-4585-42fe-abba-916ac41fcc62\") " pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:57:27.161174 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:27.160142 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d232021-de4b-4240-8877-e99fe3563101-metrics-certs\") pod \"network-metrics-daemon-l4d28\" (UID: \"5d232021-de4b-4240-8877-e99fe3563101\") " pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:57:27.161174 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:27.160184 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d7sj\" (UniqueName: \"kubernetes.io/projected/9481376a-a63f-46e2-a605-fe4bb019c331-kube-api-access-6d7sj\") pod \"network-check-target-289kb\" (UID: \"9481376a-a63f-46e2-a605-fe4bb019c331\") " pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:57:27.161174 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:27.160391 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 00:57:27.161174 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:27.160411 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 00:57:27.161174 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:27.160424 2577 projected.go:196] Error preparing data for projected volume kube-api-access-6d7sj for pod openshift-network-diagnostics/network-check-target-289kb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:57:27.161174 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:27.160475 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9481376a-a63f-46e2-a605-fe4bb019c331-kube-api-access-6d7sj podName:9481376a-a63f-46e2-a605-fe4bb019c331 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:29.16045944 +0000 UTC m=+6.072625946 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6d7sj" (UniqueName: "kubernetes.io/projected/9481376a-a63f-46e2-a605-fe4bb019c331-kube-api-access-6d7sj") pod "network-check-target-289kb" (UID: "9481376a-a63f-46e2-a605-fe4bb019c331") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:57:27.161174 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:27.160967 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 00:57:27.161174 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:27.161021 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bac70f0f-4585-42fe-abba-916ac41fcc62-original-pull-secret podName:bac70f0f-4585-42fe-abba-916ac41fcc62 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:29.161006756 +0000 UTC m=+6.073173249 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bac70f0f-4585-42fe-abba-916ac41fcc62-original-pull-secret") pod "global-pull-secret-syncer-2x7kl" (UID: "bac70f0f-4585-42fe-abba-916ac41fcc62") : object "kube-system"/"original-pull-secret" not registered Nov 04 00:57:27.161174 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:27.161107 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:57:27.161174 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:27.161152 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d232021-de4b-4240-8877-e99fe3563101-metrics-certs podName:5d232021-de4b-4240-8877-e99fe3563101 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:29.161141619 +0000 UTC m=+6.073308123 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d232021-de4b-4240-8877-e99fe3563101-metrics-certs") pod "network-metrics-daemon-l4d28" (UID: "5d232021-de4b-4240-8877-e99fe3563101") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:57:27.491055 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:27.490955 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 00:52:25 +0000 UTC" deadline="2028-08-17 09:38:33.858178882 +0000 UTC" Nov 04 00:57:27.491055 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:27.490992 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15656h41m6.367191109s" Nov 04 00:57:27.767131 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:27.767066 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:57:27.767282 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:27.767188 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2x7kl" podUID="bac70f0f-4585-42fe-abba-916ac41fcc62" Nov 04 00:57:28.766135 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:28.765256 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:57:28.766135 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:28.765589 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d28" podUID="5d232021-de4b-4240-8877-e99fe3563101" Nov 04 00:57:28.766135 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:28.765990 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:57:28.766135 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:28.766077 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289kb" podUID="9481376a-a63f-46e2-a605-fe4bb019c331" Nov 04 00:57:28.857120 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:28.857091 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-129.us-east-2.compute.internal" event={"ID":"14ca05fc22f96fd5090f819c755a4a44","Type":"ContainerStarted","Data":"d90e5e959fd335473c6b6f60266bdeb6f5524bfbc55841555fedec71947b1054"} Nov 04 00:57:29.179517 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:29.179476 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bac70f0f-4585-42fe-abba-916ac41fcc62-original-pull-secret\") pod \"global-pull-secret-syncer-2x7kl\" (UID: \"bac70f0f-4585-42fe-abba-916ac41fcc62\") " pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:57:29.179665 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:29.179527 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d232021-de4b-4240-8877-e99fe3563101-metrics-certs\") pod \"network-metrics-daemon-l4d28\" (UID: \"5d232021-de4b-4240-8877-e99fe3563101\") " pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:57:29.179665 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:29.179573 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d7sj\" (UniqueName: \"kubernetes.io/projected/9481376a-a63f-46e2-a605-fe4bb019c331-kube-api-access-6d7sj\") pod \"network-check-target-289kb\" (UID: \"9481376a-a63f-46e2-a605-fe4bb019c331\") " pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:57:29.179807 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:29.179742 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 00:57:29.179807 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:29.179761 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 00:57:29.179807 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:29.179773 2577 projected.go:196] Error preparing data for projected volume kube-api-access-6d7sj for pod openshift-network-diagnostics/network-check-target-289kb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:57:29.180007 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:29.179829 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9481376a-a63f-46e2-a605-fe4bb019c331-kube-api-access-6d7sj podName:9481376a-a63f-46e2-a605-fe4bb019c331 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:33.179811627 +0000 UTC m=+10.091978148 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6d7sj" (UniqueName: "kubernetes.io/projected/9481376a-a63f-46e2-a605-fe4bb019c331-kube-api-access-6d7sj") pod "network-check-target-289kb" (UID: "9481376a-a63f-46e2-a605-fe4bb019c331") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:57:29.180254 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:29.180232 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 00:57:29.180343 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:29.180288 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bac70f0f-4585-42fe-abba-916ac41fcc62-original-pull-secret podName:bac70f0f-4585-42fe-abba-916ac41fcc62 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:33.180273384 +0000 UTC m=+10.092439897 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bac70f0f-4585-42fe-abba-916ac41fcc62-original-pull-secret") pod "global-pull-secret-syncer-2x7kl" (UID: "bac70f0f-4585-42fe-abba-916ac41fcc62") : object "kube-system"/"original-pull-secret" not registered Nov 04 00:57:29.180402 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:29.180346 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:57:29.180402 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:29.180378 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d232021-de4b-4240-8877-e99fe3563101-metrics-certs podName:5d232021-de4b-4240-8877-e99fe3563101 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:33.180367268 +0000 UTC m=+10.092533778 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d232021-de4b-4240-8877-e99fe3563101-metrics-certs") pod "network-metrics-daemon-l4d28" (UID: "5d232021-de4b-4240-8877-e99fe3563101") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:57:29.768009 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:29.767456 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:57:29.768009 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:29.767584 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2x7kl" podUID="bac70f0f-4585-42fe-abba-916ac41fcc62" Nov 04 00:57:29.862377 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:29.862342 2577 generic.go:356] "Generic (PLEG): container finished" podID="bbfd4978-1740-4bf1-b18b-f2d7665fa421" containerID="e2ecb5bc0c15e81b9995dc5b0ff7fd6c2169588fec8f4c3688b1eea77c7fd820" exitCode=0 Nov 04 00:57:29.862512 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:29.862430 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jx882" event={"ID":"bbfd4978-1740-4bf1-b18b-f2d7665fa421","Type":"ContainerDied","Data":"e2ecb5bc0c15e81b9995dc5b0ff7fd6c2169588fec8f4c3688b1eea77c7fd820"} Nov 04 00:57:29.869356 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:29.869327 2577 generic.go:356] "Generic (PLEG): container finished" podID="12bcfb3a1cdb54b757405d90e15ad22f" containerID="9a42e94fa931a44e3e8fbe94eeadb6301397796cfbba9808d5bd07814a38e38f" exitCode=0 Nov 04 00:57:29.869491 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:29.869468 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal" event={"ID":"12bcfb3a1cdb54b757405d90e15ad22f","Type":"ContainerDied","Data":"9a42e94fa931a44e3e8fbe94eeadb6301397796cfbba9808d5bd07814a38e38f"} Nov 04 00:57:29.886683 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:29.886151 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-129.us-east-2.compute.internal" podStartSLOduration=5.886139766 podStartE2EDuration="5.886139766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:28.871982015 +0000 UTC m=+5.784148529" watchObservedRunningTime="2026-11-04 00:57:29.886139766 +0000 UTC m=+6.798306339" Nov 04 00:57:30.765526 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:30.765394 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:57:30.765690 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:30.765526 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289kb" podUID="9481376a-a63f-46e2-a605-fe4bb019c331" Nov 04 00:57:30.766080 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:30.765944 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:57:30.766080 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:30.766061 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d28" podUID="5d232021-de4b-4240-8877-e99fe3563101" Nov 04 00:57:30.877889 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:30.877264 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal" event={"ID":"12bcfb3a1cdb54b757405d90e15ad22f","Type":"ContainerStarted","Data":"b41d8409eecebc2caa2652aeff59910e560b6db64f5fec180ee58e7996cb87ba"} Nov 04 00:57:30.885233 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:30.884778 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nb955" event={"ID":"de6f6762-66c6-42c0-bf0d-7966483a040d","Type":"ContainerStarted","Data":"694521a5fb278182faf202b6efde611935c459b586807e79fe474ad1c54bb9e6"} Nov 04 00:57:30.912966 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:30.911967 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nb955" podStartSLOduration=3.860534696 podStartE2EDuration="7.911950566s" totalImagesPullingTime="4.05141587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:57:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:30.911549902 +0000 UTC m=+7.823716419" watchObservedRunningTime="2026-11-04 00:57:30.911950566 +0000 UTC m=+7.824117085" Nov 04 00:57:30.912966 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:30.912232 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal" podStartSLOduration=6.912221675 podStartE2EDuration="6.912221675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:30.893784417 +0000 UTC m=+7.805950936" watchObservedRunningTime="2026-11-04 00:57:30.912221675 +0000 UTC m=+7.824388193" Nov 04 00:57:31.767317 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:31.765564 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:57:31.767466 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:31.767359 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2x7kl" podUID="bac70f0f-4585-42fe-abba-916ac41fcc62" Nov 04 00:57:31.890540 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:31.890434 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" event={"ID":"d2344442-689b-49f6-a97b-365500025b2d","Type":"ContainerStarted","Data":"38d7e73b9854685f21efcea6d80e3186dc37d1a089a350a51679d57ec0b69600"} Nov 04 00:57:31.894482 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:31.894047 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z4snb" event={"ID":"61e0c4ee-edf6-4938-96c0-c79812e00840","Type":"ContainerStarted","Data":"a884a36fa0cc283a32921ebdaf13094391823b2cf8780f1ebb2f13fb60c4d76a"} Nov 04 00:57:31.917938 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:31.916955 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-z4snb" podStartSLOduration=3.811083801 podStartE2EDuration="8.91693985s" totalImagesPullingTime="5.105856049s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:57:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:31.916792218 +0000 UTC m=+8.828958738" watchObservedRunningTime="2026-11-04 00:57:31.91693985 +0000 UTC m=+8.829106366" Nov 04 00:57:32.766223 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:32.765657 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:57:32.766223 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:32.765768 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d28" podUID="5d232021-de4b-4240-8877-e99fe3563101" Nov 04 00:57:32.766223 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:32.766108 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:57:32.766223 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:32.766175 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289kb" podUID="9481376a-a63f-46e2-a605-fe4bb019c331" Nov 04 00:57:32.787038 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:32.787006 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:33.218386 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:33.217457 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d7sj\" (UniqueName: \"kubernetes.io/projected/9481376a-a63f-46e2-a605-fe4bb019c331-kube-api-access-6d7sj\") pod \"network-check-target-289kb\" (UID: \"9481376a-a63f-46e2-a605-fe4bb019c331\") " pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:57:33.218386 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:33.217516 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bac70f0f-4585-42fe-abba-916ac41fcc62-original-pull-secret\") pod \"global-pull-secret-syncer-2x7kl\" (UID: \"bac70f0f-4585-42fe-abba-916ac41fcc62\") " pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:57:33.218386 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:33.217547 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d232021-de4b-4240-8877-e99fe3563101-metrics-certs\") pod \"network-metrics-daemon-l4d28\" (UID: \"5d232021-de4b-4240-8877-e99fe3563101\") " pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:57:33.218386 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:33.217694 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:57:33.218386 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:33.217760 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d232021-de4b-4240-8877-e99fe3563101-metrics-certs podName:5d232021-de4b-4240-8877-e99fe3563101 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:41.217741797 +0000 UTC m=+18.129908306 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d232021-de4b-4240-8877-e99fe3563101-metrics-certs") pod "network-metrics-daemon-l4d28" (UID: "5d232021-de4b-4240-8877-e99fe3563101") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:57:33.218386 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:33.218174 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 00:57:33.218386 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:33.218194 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 00:57:33.218386 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:33.218203 2577 projected.go:196] Error preparing data for projected volume kube-api-access-6d7sj for pod openshift-network-diagnostics/network-check-target-289kb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:57:33.218386 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:33.218237 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9481376a-a63f-46e2-a605-fe4bb019c331-kube-api-access-6d7sj podName:9481376a-a63f-46e2-a605-fe4bb019c331 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:41.218226078 +0000 UTC m=+18.130392586 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6d7sj" (UniqueName: "kubernetes.io/projected/9481376a-a63f-46e2-a605-fe4bb019c331-kube-api-access-6d7sj") pod "network-check-target-289kb" (UID: "9481376a-a63f-46e2-a605-fe4bb019c331") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:57:33.218386 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:33.218284 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 00:57:33.218386 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:33.218307 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bac70f0f-4585-42fe-abba-916ac41fcc62-original-pull-secret podName:bac70f0f-4585-42fe-abba-916ac41fcc62 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:41.218300672 +0000 UTC m=+18.130467180 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bac70f0f-4585-42fe-abba-916ac41fcc62-original-pull-secret") pod "global-pull-secret-syncer-2x7kl" (UID: "bac70f0f-4585-42fe-abba-916ac41fcc62") : object "kube-system"/"original-pull-secret" not registered Nov 04 00:57:33.517986 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:33.517893 2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 04 00:57:33.767092 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:33.767007 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:57:33.768252 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:33.767989 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2x7kl" podUID="bac70f0f-4585-42fe-abba-916ac41fcc62" Nov 04 00:57:33.899791 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:33.899750 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7z64n" event={"ID":"cdcb63f9-d455-4105-8911-33dc1d3dedad","Type":"ContainerStarted","Data":"5151c9fce7fb120a5dacf44236c569279785c35aa3cbaf9bdbaa2ab4684ccc3c"} Nov 04 00:57:33.904762 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:33.904726 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" event={"ID":"d2344442-689b-49f6-a97b-365500025b2d","Type":"ContainerStarted","Data":"6072b254f590e693c803dc6aede6a39838500eb25f2465c24986ddf97bd01bbb"} Nov 04 00:57:33.924297 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:33.924246 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-7z64n" podStartSLOduration=3.052069684 podStartE2EDuration="10.924229108s" totalImagesPullingTime="7.872159424s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:57:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:33.923793849 +0000 UTC m=+10.835960368" watchObservedRunningTime="2026-11-04 00:57:33.924229108 +0000 UTC m=+10.836395626" Nov 04 00:57:34.050523 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:34.050454 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nb955" Nov 04 00:57:34.051273 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:34.051251 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nb955" Nov 04 00:57:34.503904 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:34.503816 2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-04T00:57:33.517907646Z","UUID":"6a8b87f5-4b7a-4555-9e4e-f01c4481b4b4","Handler":null,"Name":"","Endpoint":""} Nov 04 00:57:34.505279 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:34.505251 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 04 00:57:34.505417 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:34.505284 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 04 00:57:34.765256 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:34.765180 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:57:34.765256 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:34.765208 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:57:34.765441 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:34.765317 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d28" podUID="5d232021-de4b-4240-8877-e99fe3563101" Nov 04 00:57:34.765441 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:34.765424 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289kb" podUID="9481376a-a63f-46e2-a605-fe4bb019c331" Nov 04 00:57:34.909286 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:34.909253 2577 generic.go:356] "Generic (PLEG): container finished" podID="bbfd4978-1740-4bf1-b18b-f2d7665fa421" containerID="fa60e09e2c8c27515527f9822c7f7171958f7956bbec706a7fa21a64e7139845" exitCode=0 Nov 04 00:57:34.909404 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:34.909318 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jx882" event={"ID":"bbfd4978-1740-4bf1-b18b-f2d7665fa421","Type":"ContainerDied","Data":"fa60e09e2c8c27515527f9822c7f7171958f7956bbec706a7fa21a64e7139845"} Nov 04 00:57:34.913810 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:34.913787 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h4s87" event={"ID":"388103ca-b2f5-42b2-96ce-737f1ef00dc1","Type":"ContainerStarted","Data":"bd59680dc5405956a76dcdbe0401a41eafafd6b12299b7e15a3bdcf2eeb7ce5f"} Nov 04 00:57:34.951705 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:34.951660 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-h4s87" podStartSLOduration=3.75665975 podStartE2EDuration="11.951646299s" totalImagesPullingTime="8.194986549s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:57:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:34.9511009 +0000 UTC m=+11.863267418" watchObservedRunningTime="2026-11-04 00:57:34.951646299 +0000 UTC m=+11.863812820" Nov 04 00:57:35.117701 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:35.117621 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:35.259623 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:35.259560 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:35.325684 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:35.325652 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nb955" Nov 04 00:57:35.326456 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:35.326244 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nb955" Nov 04 00:57:35.466745 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:35.466503 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:35.766356 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:35.766332 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:57:35.766759 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:35.766425 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2x7kl" podUID="bac70f0f-4585-42fe-abba-916ac41fcc62" Nov 04 00:57:35.916237 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:35.916207 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-96mj9" event={"ID":"79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe","Type":"ContainerStarted","Data":"7c26f1d660dda9477284125a0a2b7bbbdbf569412d11c6e3e9e365902842befc"} Nov 04 00:57:35.917742 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:35.917714 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" event={"ID":"d8d094b5-dae6-4655-9425-c10f62f41f07","Type":"ContainerStarted","Data":"365a53e3592964a324b6c1ae0cafb660fdb42f395d4f5d15e972efb57a23cc23"} Nov 04 00:57:35.919797 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:35.919774 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" event={"ID":"d2344442-689b-49f6-a97b-365500025b2d","Type":"ContainerStarted","Data":"201168dafcdd3aeb73ff8de6380cf7117fcc1c271ceab88338a14baa23489c4f"} Nov 04 00:57:35.953609 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:35.952692 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" podStartSLOduration=2.967996617 podStartE2EDuration="12.952676501s" totalImagesPullingTime="9.984679884s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:57:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:35.952636052 +0000 UTC m=+12.864802562" watchObservedRunningTime="2026-11-04 00:57:35.952676501 +0000 UTC m=+12.864843016" Nov 04 00:57:35.953609 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:35.953349 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-96mj9" podStartSLOduration=5.162817422 podStartE2EDuration="12.953340353s" totalImagesPullingTime="7.790522931s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:57:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:35.933067621 +0000 UTC m=+12.845234139" watchObservedRunningTime="2026-11-04 00:57:35.953340353 +0000 UTC m=+12.865506870" Nov 04 00:57:36.061363 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:36.061278 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:36.204452 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:36.204420 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:36.632703 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:36.632668 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:36.764893 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:36.764863 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:57:36.765055 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:36.765018 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d28" podUID="5d232021-de4b-4240-8877-e99fe3563101" Nov 04 00:57:36.765113 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:36.765081 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:57:36.765202 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:36.765182 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289kb" podUID="9481376a-a63f-46e2-a605-fe4bb019c331" Nov 04 00:57:36.792074 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:36.792046 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:36.923416 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:36.923388 2577 generic.go:356] "Generic (PLEG): container finished" podID="bbfd4978-1740-4bf1-b18b-f2d7665fa421" containerID="7f90d5080dc5449fc083795d40bb782ef8437149ef300238b3ad23509951372d" exitCode=0 Nov 04 00:57:36.923580 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:36.923498 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jx882" event={"ID":"bbfd4978-1740-4bf1-b18b-f2d7665fa421","Type":"ContainerDied","Data":"7f90d5080dc5449fc083795d40bb782ef8437149ef300238b3ad23509951372d"} Nov 04 00:57:36.957974 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:36.957936 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" podStartSLOduration=5.356279561 podStartE2EDuration="13.957906706s" totalImagesPullingTime="8.601627145s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:57:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:35.970155055 +0000 UTC m=+12.882321709" watchObservedRunningTime="2026-11-04 00:57:36.957906706 +0000 UTC m=+13.870073220" Nov 04 00:57:37.598550 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:37.598491 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:37.765152 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:37.765122 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:57:37.765301 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:37.765243 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2x7kl" podUID="bac70f0f-4585-42fe-abba-916ac41fcc62" Nov 04 00:57:38.008652 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:38.008622 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h4s87_388103ca-b2f5-42b2-96ce-737f1ef00dc1/dns-node-resolver/0.log" Nov 04 00:57:38.201734 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:38.201711 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:38.349147 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:38.349049 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:38.765111 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:38.765081 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:57:38.765271 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:38.765221 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289kb" podUID="9481376a-a63f-46e2-a605-fe4bb019c331" Nov 04 00:57:38.765271 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:38.765269 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:57:38.765383 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:38.765363 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d28" podUID="5d232021-de4b-4240-8877-e99fe3563101" Nov 04 00:57:38.930520 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:38.930486 2577 generic.go:356] "Generic (PLEG): container finished" podID="bbfd4978-1740-4bf1-b18b-f2d7665fa421" containerID="7b744fe485874da7043a4f229d29b1f2ea6cc8abeb79da7a46a0fa496d59a53c" exitCode=0 Nov 04 00:57:38.930705 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:38.930548 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jx882" event={"ID":"bbfd4978-1740-4bf1-b18b-f2d7665fa421","Type":"ContainerDied","Data":"7b744fe485874da7043a4f229d29b1f2ea6cc8abeb79da7a46a0fa496d59a53c"} Nov 04 00:57:39.003588 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:39.003558 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:39.005861 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:39.005840 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z4snb_61e0c4ee-edf6-4938-96c0-c79812e00840/node-ca/0.log" Nov 04 00:57:39.397892 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:39.397671 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:39.658771 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:39.658673 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:39.765847 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:39.765818 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:57:39.766025 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:39.765967 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2x7kl" podUID="bac70f0f-4585-42fe-abba-916ac41fcc62" Nov 04 00:57:39.823944 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:39.823896 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:40.206623 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:40.206575 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:40.349844 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:40.349815 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:40.507806 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:40.507728 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:40.650130 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:40.650099 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:40.765281 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:40.765214 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:57:40.765423 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:40.765219 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:57:40.765423 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:40.765353 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289kb" podUID="9481376a-a63f-46e2-a605-fe4bb019c331" Nov 04 00:57:40.765423 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:40.765402 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d28" podUID="5d232021-de4b-4240-8877-e99fe3563101" Nov 04 00:57:41.050301 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:41.050238 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:41.211378 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:41.211343 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:41.282821 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:41.282790 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bac70f0f-4585-42fe-abba-916ac41fcc62-original-pull-secret\") pod \"global-pull-secret-syncer-2x7kl\" (UID: \"bac70f0f-4585-42fe-abba-916ac41fcc62\") " pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:57:41.282821 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:41.282821 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d232021-de4b-4240-8877-e99fe3563101-metrics-certs\") pod \"network-metrics-daemon-l4d28\" (UID: \"5d232021-de4b-4240-8877-e99fe3563101\") " pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:57:41.283070 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:41.282849 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d7sj\" (UniqueName: \"kubernetes.io/projected/9481376a-a63f-46e2-a605-fe4bb019c331-kube-api-access-6d7sj\") pod \"network-check-target-289kb\" (UID: \"9481376a-a63f-46e2-a605-fe4bb019c331\") " pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:57:41.283070 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:41.282957 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 00:57:41.283070 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:41.282960 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:57:41.283070 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:41.282971 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 00:57:41.283070 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:41.282988 2577 projected.go:196] Error preparing data for projected volume kube-api-access-6d7sj for pod openshift-network-diagnostics/network-check-target-289kb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:57:41.283070 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:41.282964 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 00:57:41.283070 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:41.283020 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d232021-de4b-4240-8877-e99fe3563101-metrics-certs podName:5d232021-de4b-4240-8877-e99fe3563101 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:57.28300195 +0000 UTC m=+34.195168445 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d232021-de4b-4240-8877-e99fe3563101-metrics-certs") pod "network-metrics-daemon-l4d28" (UID: "5d232021-de4b-4240-8877-e99fe3563101") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:57:41.283070 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:41.283043 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bac70f0f-4585-42fe-abba-916ac41fcc62-original-pull-secret podName:bac70f0f-4585-42fe-abba-916ac41fcc62 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:57.283026392 +0000 UTC m=+34.195192891 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bac70f0f-4585-42fe-abba-916ac41fcc62-original-pull-secret") pod "global-pull-secret-syncer-2x7kl" (UID: "bac70f0f-4585-42fe-abba-916ac41fcc62") : object "kube-system"/"original-pull-secret" not registered Nov 04 00:57:41.283070 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:41.283062 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9481376a-a63f-46e2-a605-fe4bb019c331-kube-api-access-6d7sj podName:9481376a-a63f-46e2-a605-fe4bb019c331 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:57.283052154 +0000 UTC m=+34.195218655 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6d7sj" (UniqueName: "kubernetes.io/projected/9481376a-a63f-46e2-a605-fe4bb019c331-kube-api-access-6d7sj") pod "network-check-target-289kb" (UID: "9481376a-a63f-46e2-a605-fe4bb019c331") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:57:41.765236 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:41.765211 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:57:41.765528 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:41.765351 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2x7kl" podUID="bac70f0f-4585-42fe-abba-916ac41fcc62" Nov 04 00:57:41.820206 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:41.820185 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:41.938338 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:41.938109 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" event={"ID":"26470ab5-58d3-4b17-bb82-e1b20803e7db","Type":"ContainerStarted","Data":"ff2cbee51ac3f976d7344df796e34eb555ba6be43b6176236b059cc320bdc45b"} Nov 04 00:57:41.938485 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:41.938386 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" event={"ID":"26470ab5-58d3-4b17-bb82-e1b20803e7db","Type":"ContainerStarted","Data":"6202f248918292282b3b649534fba06649e6c07d1b7a3321c4a3a0edb2de71f5"} Nov 04 00:57:41.938485 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:41.938425 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" event={"ID":"26470ab5-58d3-4b17-bb82-e1b20803e7db","Type":"ContainerStarted","Data":"eec92f84099c44d1384a2b165413a7da859b0c2db20bbedd848c6192876d66c4"} Nov 04 00:57:41.938485 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:41.938473 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" event={"ID":"26470ab5-58d3-4b17-bb82-e1b20803e7db","Type":"ContainerStarted","Data":"9c103d2f30eed6d5b7e85b31ccf5c7d1d332669e1343dd0f24a1337a47eb7b28"} Nov 04 00:57:41.938606 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:41.938514 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" event={"ID":"26470ab5-58d3-4b17-bb82-e1b20803e7db","Type":"ContainerStarted","Data":"5380fbca9e0048b7e2042aada162ee19620dbf9e34d5479d1992ed522638eeb4"} Nov 04 00:57:41.938606 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:41.938557 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" event={"ID":"26470ab5-58d3-4b17-bb82-e1b20803e7db","Type":"ContainerStarted","Data":"e9aad816950cb1e31475ae116cf26e0bb5ae07bc31dd4a636f788e59526d2e43"} Nov 04 00:57:41.940853 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:41.940833 2577 generic.go:356] "Generic (PLEG): container finished" podID="bbfd4978-1740-4bf1-b18b-f2d7665fa421" containerID="ebb05f5e443426872a507070cb1c56b879882b49d79a07c47cf523ea91c11e77" exitCode=0 Nov 04 00:57:41.940946 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:41.940863 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jx882" event={"ID":"bbfd4978-1740-4bf1-b18b-f2d7665fa421","Type":"ContainerDied","Data":"ebb05f5e443426872a507070cb1c56b879882b49d79a07c47cf523ea91c11e77"} Nov 04 00:57:41.972540 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:41.972518 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:42.409067 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:42.409034 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:42.613335 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:42.613307 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:42.765564 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:42.765542 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:57:42.765888 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:42.765638 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d28" podUID="5d232021-de4b-4240-8877-e99fe3563101" Nov 04 00:57:42.765888 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:42.765662 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:57:42.765888 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:42.765701 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289kb" podUID="9481376a-a63f-46e2-a605-fe4bb019c331" Nov 04 00:57:42.946011 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:42.945985 2577 generic.go:356] "Generic (PLEG): container finished" podID="bbfd4978-1740-4bf1-b18b-f2d7665fa421" containerID="01fa48d2462f5916942ddb588d8e0e3bd19922228f64799346e44531e2e9e20d" exitCode=0 Nov 04 00:57:42.946124 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:42.946042 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jx882" event={"ID":"bbfd4978-1740-4bf1-b18b-f2d7665fa421","Type":"ContainerDied","Data":"01fa48d2462f5916942ddb588d8e0e3bd19922228f64799346e44531e2e9e20d"} Nov 04 00:57:43.038948 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:43.038907 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:43.184702 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:43.184675 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:43.325396 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:43.325338 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:43.470413 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:43.470385 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:43.629574 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:43.629523 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:43.761193 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:43.761165 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:43.765863 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:43.765846 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:57:43.766233 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:43.765918 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2x7kl" podUID="bac70f0f-4585-42fe-abba-916ac41fcc62" Nov 04 00:57:43.893914 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:43.893890 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:43.951797 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:43.951763 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" event={"ID":"26470ab5-58d3-4b17-bb82-e1b20803e7db","Type":"ContainerStarted","Data":"152371a7c87272bf77939a691f6593105290178c3f21283b6f2dc5acb5c48ae7"} Nov 04 00:57:43.955102 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:43.955082 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jx882" event={"ID":"bbfd4978-1740-4bf1-b18b-f2d7665fa421","Type":"ContainerStarted","Data":"ad19e92cffb392e645d212fbf81dd2ba549c2b4cf0f5b5541ad06ac88d2f894d"} Nov 04 00:57:43.979192 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:43.979153 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jx882" podStartSLOduration=9.413708481 podStartE2EDuration="20.979143292s" totalImagesPullingTime="11.565434811s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:57:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:43.979010395 +0000 UTC m=+20.891176912" watchObservedRunningTime="2026-11-04 00:57:43.979143292 +0000 UTC m=+20.891309809" Nov 04 00:57:44.398050 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:44.398023 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:44.527003 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:44.526974 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:44.662180 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:44.662119 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:44.765240 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:44.765219 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:57:44.765324 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:44.765247 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:57:44.765370 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:44.765318 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d28" podUID="5d232021-de4b-4240-8877-e99fe3563101" Nov 04 00:57:44.765411 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:44.765382 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289kb" podUID="9481376a-a63f-46e2-a605-fe4bb019c331" Nov 04 00:57:44.900603 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:44.900582 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:45.034110 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:45.034079 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:45.212146 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:45.212119 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:45.340725 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:45.340670 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:45.474584 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:45.474555 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:45.614868 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:45.614814 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:45.748086 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:45.748059 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:45.764900 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:45.764878 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:57:45.765041 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:45.765006 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2x7kl" podUID="bac70f0f-4585-42fe-abba-916ac41fcc62" Nov 04 00:57:46.397354 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:46.397273 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:46.765744 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:46.765715 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:57:46.765744 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:46.765735 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:57:46.765937 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:46.765833 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d28" podUID="5d232021-de4b-4240-8877-e99fe3563101" Nov 04 00:57:46.766003 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:46.765983 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289kb" podUID="9481376a-a63f-46e2-a605-fe4bb019c331" Nov 04 00:57:46.806627 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:46.806604 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:46.965473 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:46.965447 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" event={"ID":"26470ab5-58d3-4b17-bb82-e1b20803e7db","Type":"ContainerStarted","Data":"914548068e3cc94a9cc60e3c0c40e4bf04c0f4d14e20289536534c823a606806"} Nov 04 00:57:46.965755 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:46.965737 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:46.965842 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:46.965814 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:46.982276 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:46.982254 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:46.995193 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:46.995157 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" podStartSLOduration=8.269433019 podStartE2EDuration="23.995146304s" totalImagesPullingTime="15.725713285s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:57:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:46.99483207 +0000 UTC m=+23.906998597" watchObservedRunningTime="2026-11-04 00:57:46.995146304 +0000 UTC m=+23.907312820" Nov 04 00:57:46.998037 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:46.998012 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:47.197864 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:47.197842 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:47.599353 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:47.599291 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:47.764769 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:47.764736 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:57:47.764898 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:47.764839 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2x7kl" podUID="bac70f0f-4585-42fe-abba-916ac41fcc62" Nov 04 00:57:47.967659 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:47.967634 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:47.984332 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:47.984311 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:57:48.415666 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:48.415588 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:48.765017 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:48.764989 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:57:48.765434 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:48.765078 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289kb" podUID="9481376a-a63f-46e2-a605-fe4bb019c331" Nov 04 00:57:48.765434 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:48.765133 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:57:48.765434 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:48.765217 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d28" podUID="5d232021-de4b-4240-8877-e99fe3563101" Nov 04 00:57:48.796840 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:48.796813 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:48.991494 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:48.991466 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:49.195975 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:49.195945 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:49.391912 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:49.391884 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:49.764941 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:49.764901 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:57:49.765092 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:49.765027 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2x7kl" podUID="bac70f0f-4585-42fe-abba-916ac41fcc62" Nov 04 00:57:50.765265 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:50.765233 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:57:50.765725 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:50.765246 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:57:50.765725 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:50.765341 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289kb" podUID="9481376a-a63f-46e2-a605-fe4bb019c331" Nov 04 00:57:50.765725 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:50.765413 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d28" podUID="5d232021-de4b-4240-8877-e99fe3563101" Nov 04 00:57:51.331765 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:51.331733 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:51.465180 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:51.465146 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:51.764745 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:51.764713 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:57:51.764901 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:51.764830 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2x7kl" podUID="bac70f0f-4585-42fe-abba-916ac41fcc62" Nov 04 00:57:52.764662 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:52.764633 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:57:52.765036 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:52.764633 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:57:52.765036 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:52.764733 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289kb" podUID="9481376a-a63f-46e2-a605-fe4bb019c331" Nov 04 00:57:52.765036 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:52.764811 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d28" podUID="5d232021-de4b-4240-8877-e99fe3563101" Nov 04 00:57:53.755301 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:53.755125 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:53.765520 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:53.765500 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:57:53.765864 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:53.765574 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2x7kl" podUID="bac70f0f-4585-42fe-abba-916ac41fcc62" Nov 04 00:57:53.903787 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:53.903764 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:54.765706 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:54.765677 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:57:54.766163 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:54.765801 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d28" podUID="5d232021-de4b-4240-8877-e99fe3563101" Nov 04 00:57:54.766163 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:54.765849 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:57:54.766163 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:54.765952 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289kb" podUID="9481376a-a63f-46e2-a605-fe4bb019c331" Nov 04 00:57:55.585296 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:55.585269 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:55.765317 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:55.765284 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:57:55.765470 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:55.765416 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2x7kl" podUID="bac70f0f-4585-42fe-abba-916ac41fcc62" Nov 04 00:57:55.769565 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:55.769520 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:56.764971 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:56.764910 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:57:56.765212 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:56.764973 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:57:56.765212 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:56.765084 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d28" podUID="5d232021-de4b-4240-8877-e99fe3563101" Nov 04 00:57:56.765524 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:56.765496 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289kb" podUID="9481376a-a63f-46e2-a605-fe4bb019c331" Nov 04 00:57:56.823553 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:56.823459 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2x7kl"] Nov 04 00:57:56.823877 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:56.823557 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:57:56.823877 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:56.823637 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2x7kl" podUID="bac70f0f-4585-42fe-abba-916ac41fcc62" Nov 04 00:57:56.825654 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:56.825629 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-289kb"] Nov 04 00:57:56.828874 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:56.828810 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l4d28"] Nov 04 00:57:56.916160 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:56.916140 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:56.984023 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:56.984001 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:57:56.984137 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:56.984012 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:57:56.984137 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:56.984101 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d28" podUID="5d232021-de4b-4240-8877-e99fe3563101" Nov 04 00:57:56.984206 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:56.984166 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289kb" podUID="9481376a-a63f-46e2-a605-fe4bb019c331" Nov 04 00:57:57.064767 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:57.064743 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:57.193750 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:57.193728 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h4s87_388103ca-b2f5-42b2-96ce-737f1ef00dc1/dns-node-resolver/0.log" Nov 04 00:57:57.293824 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:57.293799 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d7sj\" (UniqueName: \"kubernetes.io/projected/9481376a-a63f-46e2-a605-fe4bb019c331-kube-api-access-6d7sj\") pod \"network-check-target-289kb\" (UID: \"9481376a-a63f-46e2-a605-fe4bb019c331\") " pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:57:57.293954 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:57.293839 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bac70f0f-4585-42fe-abba-916ac41fcc62-original-pull-secret\") pod \"global-pull-secret-syncer-2x7kl\" (UID: \"bac70f0f-4585-42fe-abba-916ac41fcc62\") " pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:57:57.293954 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:57.293855 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d232021-de4b-4240-8877-e99fe3563101-metrics-certs\") pod \"network-metrics-daemon-l4d28\" (UID: \"5d232021-de4b-4240-8877-e99fe3563101\") " pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:57:57.293954 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:57.293951 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:57:57.294110 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:57.293963 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 00:57:57.294110 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:57.293970 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 00:57:57.294110 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:57.293978 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 00:57:57.294110 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:57.293994 2577 projected.go:196] Error preparing data for projected volume kube-api-access-6d7sj for pod openshift-network-diagnostics/network-check-target-289kb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:57:57.294110 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:57.293996 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d232021-de4b-4240-8877-e99fe3563101-metrics-certs podName:5d232021-de4b-4240-8877-e99fe3563101 nodeName:}" failed. No retries permitted until 2026-11-04 00:58:29.293983084 +0000 UTC m=+66.206149579 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d232021-de4b-4240-8877-e99fe3563101-metrics-certs") pod "network-metrics-daemon-l4d28" (UID: "5d232021-de4b-4240-8877-e99fe3563101") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:57:57.294110 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:57.294041 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bac70f0f-4585-42fe-abba-916ac41fcc62-original-pull-secret podName:bac70f0f-4585-42fe-abba-916ac41fcc62 nodeName:}" failed. No retries permitted until 2026-11-04 00:58:29.294024176 +0000 UTC m=+66.206190689 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bac70f0f-4585-42fe-abba-916ac41fcc62-original-pull-secret") pod "global-pull-secret-syncer-2x7kl" (UID: "bac70f0f-4585-42fe-abba-916ac41fcc62") : object "kube-system"/"original-pull-secret" not registered Nov 04 00:57:57.294110 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:57.294058 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9481376a-a63f-46e2-a605-fe4bb019c331-kube-api-access-6d7sj podName:9481376a-a63f-46e2-a605-fe4bb019c331 nodeName:}" failed. No retries permitted until 2026-11-04 00:58:29.294048611 +0000 UTC m=+66.206215106 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6d7sj" (UniqueName: "kubernetes.io/projected/9481376a-a63f-46e2-a605-fe4bb019c331-kube-api-access-6d7sj") pod "network-check-target-289kb" (UID: "9481376a-a63f-46e2-a605-fe4bb019c331") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:57:57.504571 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:57.504507 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:57.665799 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:57.665761 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:57.807784 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:57.807713 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:57.939796 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:57.939772 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:58.393779 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:58.393713 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z4snb_61e0c4ee-edf6-4938-96c0-c79812e00840/node-ca/0.log" Nov 04 00:57:58.727097 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:58.727071 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:58.765120 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:58.765096 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:57:58.765222 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:58.765199 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289kb" podUID="9481376a-a63f-46e2-a605-fe4bb019c331" Nov 04 00:57:58.765285 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:58.765272 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:57:58.765373 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:58.765358 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2x7kl" podUID="bac70f0f-4585-42fe-abba-916ac41fcc62" Nov 04 00:57:58.765415 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:58.765368 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:57:58.765475 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:57:58.765458 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d28" podUID="5d232021-de4b-4240-8877-e99fe3563101" Nov 04 00:57:58.869107 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:58.869072 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:59.449383 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:59.449344 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:59.584171 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:59.584134 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:59.723227 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:57:59.723154 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:00.765384 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:00.765352 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:58:00.765384 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:00.765383 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:58:00.765889 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:58:00.765459 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-289kb" podUID="9481376a-a63f-46e2-a605-fe4bb019c331" Nov 04 00:58:00.765889 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:00.765510 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:58:00.765889 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:58:00.765620 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2x7kl" podUID="bac70f0f-4585-42fe-abba-916ac41fcc62" Nov 04 00:58:00.765889 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:58:00.765718 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d28" podUID="5d232021-de4b-4240-8877-e99fe3563101" Nov 04 00:58:01.146784 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.146751 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:01.279333 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.279305 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:01.361861 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.361806 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-129.us-east-2.compute.internal" event="NodeReady" Nov 04 00:58:01.361980 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.361900 2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 04 00:58:01.419152 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.419129 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hhndm"] Nov 04 00:58:01.452036 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.452016 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7jm5h"] Nov 04 00:58:01.452153 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.452137 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hhndm" Nov 04 00:58:01.456190 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.456168 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 04 00:58:01.456326 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.456193 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cjss9\"" Nov 04 00:58:01.456766 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.456746 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 04 00:58:01.468115 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.468092 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hhndm"] Nov 04 00:58:01.468209 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.468135 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7jm5h"] Nov 04 00:58:01.468209 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.468169 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cv4w2"] Nov 04 00:58:01.468209 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.468194 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7jm5h" Nov 04 00:58:01.482145 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.482128 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cv4w2"] Nov 04 00:58:01.482250 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.482233 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cv4w2" Nov 04 00:58:01.528045 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.528025 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d08dca6d-7563-4d2c-ae92-e7ffd4beda06-config-volume\") pod \"dns-default-hhndm\" (UID: \"d08dca6d-7563-4d2c-ae92-e7ffd4beda06\") " pod="openshift-dns/dns-default-hhndm" Nov 04 00:58:01.528142 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.528052 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d08dca6d-7563-4d2c-ae92-e7ffd4beda06-metrics-tls\") pod \"dns-default-hhndm\" (UID: \"d08dca6d-7563-4d2c-ae92-e7ffd4beda06\") " pod="openshift-dns/dns-default-hhndm" Nov 04 00:58:01.528142 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.528070 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59e9f4f9-590f-43b0-999d-1fad31779d16-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7jm5h\" (UID: \"59e9f4f9-590f-43b0-999d-1fad31779d16\") " pod="openshift-insights/insights-runtime-extractor-7jm5h" Nov 04 00:58:01.528242 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.528135 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59e9f4f9-590f-43b0-999d-1fad31779d16-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7jm5h\" (UID: \"59e9f4f9-590f-43b0-999d-1fad31779d16\") " pod="openshift-insights/insights-runtime-extractor-7jm5h" Nov 04 00:58:01.528297 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.528256 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d08dca6d-7563-4d2c-ae92-e7ffd4beda06-tmp-dir\") pod \"dns-default-hhndm\" (UID: \"d08dca6d-7563-4d2c-ae92-e7ffd4beda06\") " pod="openshift-dns/dns-default-hhndm" Nov 04 00:58:01.528336 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.528297 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znlq8\" (UniqueName: \"kubernetes.io/projected/d08dca6d-7563-4d2c-ae92-e7ffd4beda06-kube-api-access-znlq8\") pod \"dns-default-hhndm\" (UID: \"d08dca6d-7563-4d2c-ae92-e7ffd4beda06\") " pod="openshift-dns/dns-default-hhndm" Nov 04 00:58:01.528336 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.528327 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59e9f4f9-590f-43b0-999d-1fad31779d16-data-volume\") pod \"insights-runtime-extractor-7jm5h\" (UID: \"59e9f4f9-590f-43b0-999d-1fad31779d16\") " pod="openshift-insights/insights-runtime-extractor-7jm5h" Nov 04 00:58:01.528424 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.528388 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59e9f4f9-590f-43b0-999d-1fad31779d16-crio-socket\") pod \"insights-runtime-extractor-7jm5h\" (UID: \"59e9f4f9-590f-43b0-999d-1fad31779d16\") " pod="openshift-insights/insights-runtime-extractor-7jm5h" Nov 04 00:58:01.528473 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.528420 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76469\" (UniqueName: \"kubernetes.io/projected/59e9f4f9-590f-43b0-999d-1fad31779d16-kube-api-access-76469\") pod \"insights-runtime-extractor-7jm5h\" (UID: \"59e9f4f9-590f-43b0-999d-1fad31779d16\") " pod="openshift-insights/insights-runtime-extractor-7jm5h" Nov 04 00:58:01.556157 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.556134 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 04 00:58:01.556240 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.556224 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ldjm6\"" Nov 04 00:58:01.556296 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.556280 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-pb69q\"" Nov 04 00:58:01.556363 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.556347 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 04 00:58:01.556489 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.556478 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 04 00:58:01.557594 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.557580 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 04 00:58:01.557627 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.557580 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 04 00:58:01.557627 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.557613 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 04 00:58:01.557686 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.557630 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 04 00:58:01.629091 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.629040 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d08dca6d-7563-4d2c-ae92-e7ffd4beda06-config-volume\") pod \"dns-default-hhndm\" (UID: \"d08dca6d-7563-4d2c-ae92-e7ffd4beda06\") " pod="openshift-dns/dns-default-hhndm" Nov 04 00:58:01.629091 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.629067 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d08dca6d-7563-4d2c-ae92-e7ffd4beda06-metrics-tls\") pod \"dns-default-hhndm\" (UID: \"d08dca6d-7563-4d2c-ae92-e7ffd4beda06\") " pod="openshift-dns/dns-default-hhndm" Nov 04 00:58:01.629091 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.629083 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59e9f4f9-590f-43b0-999d-1fad31779d16-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7jm5h\" (UID: \"59e9f4f9-590f-43b0-999d-1fad31779d16\") " pod="openshift-insights/insights-runtime-extractor-7jm5h" Nov 04 00:58:01.629292 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.629109 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d90c5697-ccdd-4845-8972-dbbd952dcc87-cert\") pod \"ingress-canary-cv4w2\" (UID: \"d90c5697-ccdd-4845-8972-dbbd952dcc87\") " pod="openshift-ingress-canary/ingress-canary-cv4w2" Nov 04 00:58:01.629292 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.629160 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zrwn\" (UniqueName: \"kubernetes.io/projected/d90c5697-ccdd-4845-8972-dbbd952dcc87-kube-api-access-2zrwn\") pod \"ingress-canary-cv4w2\" (UID: \"d90c5697-ccdd-4845-8972-dbbd952dcc87\") " pod="openshift-ingress-canary/ingress-canary-cv4w2" Nov 04 00:58:01.629292 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.629269 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59e9f4f9-590f-43b0-999d-1fad31779d16-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7jm5h\" (UID: \"59e9f4f9-590f-43b0-999d-1fad31779d16\") " pod="openshift-insights/insights-runtime-extractor-7jm5h" Nov 04 00:58:01.629292 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.629291 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d08dca6d-7563-4d2c-ae92-e7ffd4beda06-tmp-dir\") pod \"dns-default-hhndm\" (UID: \"d08dca6d-7563-4d2c-ae92-e7ffd4beda06\") " pod="openshift-dns/dns-default-hhndm" Nov 04 00:58:01.629476 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.629306 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znlq8\" (UniqueName: \"kubernetes.io/projected/d08dca6d-7563-4d2c-ae92-e7ffd4beda06-kube-api-access-znlq8\") pod \"dns-default-hhndm\" (UID: \"d08dca6d-7563-4d2c-ae92-e7ffd4beda06\") " pod="openshift-dns/dns-default-hhndm" Nov 04 00:58:01.629476 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.629321 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59e9f4f9-590f-43b0-999d-1fad31779d16-data-volume\") pod \"insights-runtime-extractor-7jm5h\" (UID: \"59e9f4f9-590f-43b0-999d-1fad31779d16\") " pod="openshift-insights/insights-runtime-extractor-7jm5h" Nov 04 00:58:01.629476 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.629337 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59e9f4f9-590f-43b0-999d-1fad31779d16-crio-socket\") pod \"insights-runtime-extractor-7jm5h\" (UID: \"59e9f4f9-590f-43b0-999d-1fad31779d16\") " pod="openshift-insights/insights-runtime-extractor-7jm5h" Nov 04 00:58:01.629476 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.629359 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76469\" (UniqueName: \"kubernetes.io/projected/59e9f4f9-590f-43b0-999d-1fad31779d16-kube-api-access-76469\") pod \"insights-runtime-extractor-7jm5h\" (UID: \"59e9f4f9-590f-43b0-999d-1fad31779d16\") " pod="openshift-insights/insights-runtime-extractor-7jm5h" Nov 04 00:58:01.629666 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.629595 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d08dca6d-7563-4d2c-ae92-e7ffd4beda06-tmp-dir\") pod \"dns-default-hhndm\" (UID: \"d08dca6d-7563-4d2c-ae92-e7ffd4beda06\") " pod="openshift-dns/dns-default-hhndm" Nov 04 00:58:01.629721 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.629663 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59e9f4f9-590f-43b0-999d-1fad31779d16-data-volume\") pod \"insights-runtime-extractor-7jm5h\" (UID: \"59e9f4f9-590f-43b0-999d-1fad31779d16\") " pod="openshift-insights/insights-runtime-extractor-7jm5h" Nov 04 00:58:01.629840 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.629822 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d08dca6d-7563-4d2c-ae92-e7ffd4beda06-config-volume\") pod \"dns-default-hhndm\" (UID: \"d08dca6d-7563-4d2c-ae92-e7ffd4beda06\") " pod="openshift-dns/dns-default-hhndm" Nov 04 00:58:01.629840 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.629829 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59e9f4f9-590f-43b0-999d-1fad31779d16-crio-socket\") pod \"insights-runtime-extractor-7jm5h\" (UID: \"59e9f4f9-590f-43b0-999d-1fad31779d16\") " pod="openshift-insights/insights-runtime-extractor-7jm5h" Nov 04 00:58:01.629948 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.629835 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59e9f4f9-590f-43b0-999d-1fad31779d16-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7jm5h\" (UID: \"59e9f4f9-590f-43b0-999d-1fad31779d16\") " pod="openshift-insights/insights-runtime-extractor-7jm5h" Nov 04 00:58:01.633467 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.633372 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d08dca6d-7563-4d2c-ae92-e7ffd4beda06-metrics-tls\") pod \"dns-default-hhndm\" (UID: \"d08dca6d-7563-4d2c-ae92-e7ffd4beda06\") " pod="openshift-dns/dns-default-hhndm" Nov 04 00:58:01.633514 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.633427 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59e9f4f9-590f-43b0-999d-1fad31779d16-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7jm5h\" (UID: \"59e9f4f9-590f-43b0-999d-1fad31779d16\") " pod="openshift-insights/insights-runtime-extractor-7jm5h" Nov 04 00:58:01.637395 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.637372 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znlq8\" (UniqueName: \"kubernetes.io/projected/d08dca6d-7563-4d2c-ae92-e7ffd4beda06-kube-api-access-znlq8\") pod \"dns-default-hhndm\" (UID: \"d08dca6d-7563-4d2c-ae92-e7ffd4beda06\") " pod="openshift-dns/dns-default-hhndm" Nov 04 00:58:01.637493 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.637430 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76469\" (UniqueName: \"kubernetes.io/projected/59e9f4f9-590f-43b0-999d-1fad31779d16-kube-api-access-76469\") pod \"insights-runtime-extractor-7jm5h\" (UID: \"59e9f4f9-590f-43b0-999d-1fad31779d16\") " pod="openshift-insights/insights-runtime-extractor-7jm5h" Nov 04 00:58:01.730487 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.730465 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d90c5697-ccdd-4845-8972-dbbd952dcc87-cert\") pod \"ingress-canary-cv4w2\" (UID: \"d90c5697-ccdd-4845-8972-dbbd952dcc87\") " pod="openshift-ingress-canary/ingress-canary-cv4w2" Nov 04 00:58:01.730590 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.730500 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zrwn\" (UniqueName: \"kubernetes.io/projected/d90c5697-ccdd-4845-8972-dbbd952dcc87-kube-api-access-2zrwn\") pod \"ingress-canary-cv4w2\" (UID: \"d90c5697-ccdd-4845-8972-dbbd952dcc87\") " pod="openshift-ingress-canary/ingress-canary-cv4w2" Nov 04 00:58:01.732601 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.732577 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d90c5697-ccdd-4845-8972-dbbd952dcc87-cert\") pod \"ingress-canary-cv4w2\" (UID: \"d90c5697-ccdd-4845-8972-dbbd952dcc87\") " pod="openshift-ingress-canary/ingress-canary-cv4w2" Nov 04 00:58:01.738978 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.738959 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zrwn\" (UniqueName: \"kubernetes.io/projected/d90c5697-ccdd-4845-8972-dbbd952dcc87-kube-api-access-2zrwn\") pod \"ingress-canary-cv4w2\" (UID: \"d90c5697-ccdd-4845-8972-dbbd952dcc87\") " pod="openshift-ingress-canary/ingress-canary-cv4w2" Nov 04 00:58:01.788762 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.788740 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hhndm" Nov 04 00:58:01.830044 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.830024 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7jm5h" Nov 04 00:58:01.843843 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:01.843811 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cv4w2" Nov 04 00:58:02.070526 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:02.070496 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:02.143356 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:02.143329 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cv4w2"] Nov 04 00:58:02.146525 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:58:02.146495 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd90c5697_ccdd_4845_8972_dbbd952dcc87.slice/crio-288b26a3d92fef361cf864edee46bc80d706a3065e7800ba294dcfc9d2c80a80 WatchSource:0}: Error finding container 288b26a3d92fef361cf864edee46bc80d706a3065e7800ba294dcfc9d2c80a80: Status 404 returned error can't find the container with id 288b26a3d92fef361cf864edee46bc80d706a3065e7800ba294dcfc9d2c80a80 Nov 04 00:58:02.147572 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:02.147549 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hhndm"] Nov 04 00:58:02.148271 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:02.148252 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7jm5h"] Nov 04 00:58:02.149677 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:58:02.149647 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59e9f4f9_590f_43b0_999d_1fad31779d16.slice/crio-5f51c29e18d6e587b4eaec9f70a10e1f69be2eb1842d5125c4dae0ef0d221f66 WatchSource:0}: Error finding container 5f51c29e18d6e587b4eaec9f70a10e1f69be2eb1842d5125c4dae0ef0d221f66: Status 404 returned error can't find the container with id 5f51c29e18d6e587b4eaec9f70a10e1f69be2eb1842d5125c4dae0ef0d221f66 Nov 04 00:58:02.150806 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:58:02.150568 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd08dca6d_7563_4d2c_ae92_e7ffd4beda06.slice/crio-2d6a85bdb7f1cc7f2e3c02ac1a89a9769dc55dc1bafd34d61098ba3cde6da252 WatchSource:0}: Error finding container 2d6a85bdb7f1cc7f2e3c02ac1a89a9769dc55dc1bafd34d61098ba3cde6da252: Status 404 returned error can't find the container with id 2d6a85bdb7f1cc7f2e3c02ac1a89a9769dc55dc1bafd34d61098ba3cde6da252 Nov 04 00:58:02.211278 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:02.211258 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:02.764762 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:02.764731 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:58:02.764985 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:02.764837 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:58:02.764985 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:02.764850 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:58:02.769572 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:02.769551 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 04 00:58:02.771209 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:02.771186 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-nszlw\"" Nov 04 00:58:02.771209 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:02.771208 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 04 00:58:02.771443 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:02.771271 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4rmxc\"" Nov 04 00:58:02.771443 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:02.771347 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q48lg\"" Nov 04 00:58:02.772653 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:02.772470 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 00:58:02.772653 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:02.772601 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 00:58:02.999524 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:02.999490 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cv4w2" event={"ID":"d90c5697-ccdd-4845-8972-dbbd952dcc87","Type":"ContainerStarted","Data":"288b26a3d92fef361cf864edee46bc80d706a3065e7800ba294dcfc9d2c80a80"} Nov 04 00:58:03.001368 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:03.001341 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7jm5h" event={"ID":"59e9f4f9-590f-43b0-999d-1fad31779d16","Type":"ContainerStarted","Data":"a8f0b50b4535308f49deb2a207e60cdb23284edbb8f4c1c90ed5d2274a6ef485"} Nov 04 00:58:03.001506 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:03.001398 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7jm5h" event={"ID":"59e9f4f9-590f-43b0-999d-1fad31779d16","Type":"ContainerStarted","Data":"5f51c29e18d6e587b4eaec9f70a10e1f69be2eb1842d5125c4dae0ef0d221f66"} Nov 04 00:58:03.002811 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:03.002782 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hhndm" event={"ID":"d08dca6d-7563-4d2c-ae92-e7ffd4beda06","Type":"ContainerStarted","Data":"2d6a85bdb7f1cc7f2e3c02ac1a89a9769dc55dc1bafd34d61098ba3cde6da252"} Nov 04 00:58:03.965659 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:03.965469 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:04.008822 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:04.008782 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7jm5h" event={"ID":"59e9f4f9-590f-43b0-999d-1fad31779d16","Type":"ContainerStarted","Data":"56c7287d82d5ff9ea49bb38da8d80ab8199b3ff0f1d74bcac59977d017bd18d0"} Nov 04 00:58:04.683760 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:04.683730 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:04.865338 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:04.865307 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:04.889818 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:04.888664 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-dl2dc"] Nov 04 00:58:04.909557 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:04.909536 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dl2dc" Nov 04 00:58:04.922456 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:04.922427 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 04 00:58:04.923392 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:04.923367 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-dl2dc"] Nov 04 00:58:04.923691 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:04.923667 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 04 00:58:04.923771 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:04.923741 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 04 00:58:04.923771 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:04.923754 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 04 00:58:04.924474 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:04.924454 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xz2sx\"" Nov 04 00:58:04.924565 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:04.924482 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 04 00:58:04.938590 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:04.938560 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25"] Nov 04 00:58:04.951273 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:04.951248 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" Nov 04 00:58:04.956775 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:04.956178 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mhb4l"] Nov 04 00:58:04.970317 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:04.970292 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25"] Nov 04 00:58:04.970418 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:04.970407 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:05.021616 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.021584 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 04 00:58:05.023217 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.023192 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 04 00:58:05.023565 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.023527 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 04 00:58:05.023725 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.023710 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 04 00:58:05.024503 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.024457 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 04 00:58:05.024503 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.024459 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 04 00:58:05.024694 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.024507 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-gtbz7\"" Nov 04 00:58:05.024694 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.024566 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5llll\"" Nov 04 00:58:05.036194 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.036161 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hhndm" event={"ID":"d08dca6d-7563-4d2c-ae92-e7ffd4beda06","Type":"ContainerStarted","Data":"edb41d4eff967c2d1183f93b8a9d045d1f3dfe317a6324e8debb9b40559625a1"} Nov 04 00:58:05.036286 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.036231 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hhndm" event={"ID":"d08dca6d-7563-4d2c-ae92-e7ffd4beda06","Type":"ContainerStarted","Data":"7eb61a9a405189882e1b38d344919c241e14c418d8ce4bf51206aa05cb26ada6"} Nov 04 00:58:05.049142 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.049122 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hhndm" Nov 04 00:58:05.056094 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.056071 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/95638663-fffb-41a6-8056-c1f4293a3fb0-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-8wz25\" (UID: \"95638663-fffb-41a6-8056-c1f4293a3fb0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" Nov 04 00:58:05.056209 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.056107 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr5ms\" (UniqueName: \"kubernetes.io/projected/1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84-kube-api-access-wr5ms\") pod \"openshift-state-metrics-7bd7489957-dl2dc\" (UID: \"1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dl2dc" Nov 04 00:58:05.056209 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.056136 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95638663-fffb-41a6-8056-c1f4293a3fb0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-8wz25\" (UID: \"95638663-fffb-41a6-8056-c1f4293a3fb0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" Nov 04 00:58:05.056209 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.056160 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5522ff12-fed1-4411-96b0-1cceb6f0b634-node-exporter-wtmp\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:05.056209 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.056179 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-dl2dc\" (UID: \"1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dl2dc" Nov 04 00:58:05.056209 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.056200 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5522ff12-fed1-4411-96b0-1cceb6f0b634-node-exporter-accelerators-collector-config\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:05.056452 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.056254 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5522ff12-fed1-4411-96b0-1cceb6f0b634-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:05.056452 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.056278 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/95638663-fffb-41a6-8056-c1f4293a3fb0-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-8wz25\" (UID: \"95638663-fffb-41a6-8056-c1f4293a3fb0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" Nov 04 00:58:05.056452 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.056307 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/95638663-fffb-41a6-8056-c1f4293a3fb0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-8wz25\" (UID: \"95638663-fffb-41a6-8056-c1f4293a3fb0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" Nov 04 00:58:05.056452 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.056336 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-dl2dc\" (UID: \"1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dl2dc" Nov 04 00:58:05.056452 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.056377 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95638663-fffb-41a6-8056-c1f4293a3fb0-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-8wz25\" (UID: \"95638663-fffb-41a6-8056-c1f4293a3fb0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" Nov 04 00:58:05.056452 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.056399 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-dl2dc\" (UID: \"1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dl2dc" Nov 04 00:58:05.056452 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.056443 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5522ff12-fed1-4411-96b0-1cceb6f0b634-sys\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:05.056666 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.056496 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5522ff12-fed1-4411-96b0-1cceb6f0b634-node-exporter-textfile\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:05.056666 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.056521 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5522ff12-fed1-4411-96b0-1cceb6f0b634-node-exporter-tls\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:05.056666 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.056544 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5522ff12-fed1-4411-96b0-1cceb6f0b634-metrics-client-ca\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:05.056666 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.056573 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5522ff12-fed1-4411-96b0-1cceb6f0b634-root\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:05.056666 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.056597 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59m4r\" (UniqueName: \"kubernetes.io/projected/5522ff12-fed1-4411-96b0-1cceb6f0b634-kube-api-access-59m4r\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:05.056666 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.056634 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqzsj\" (UniqueName: \"kubernetes.io/projected/95638663-fffb-41a6-8056-c1f4293a3fb0-kube-api-access-hqzsj\") pod \"kube-state-metrics-5dc8d7d89b-8wz25\" (UID: \"95638663-fffb-41a6-8056-c1f4293a3fb0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" Nov 04 00:58:05.076954 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.076892 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-hhndm" podStartSLOduration=1.552289048 podStartE2EDuration="4.076880255s" totalImagesPullingTime="2.524591207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:58:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:05.076005587 +0000 UTC m=+41.988172106" watchObservedRunningTime="2026-11-04 00:58:05.076880255 +0000 UTC m=+41.989046773" Nov 04 00:58:05.097677 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.097584 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:05.157691 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.157654 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5522ff12-fed1-4411-96b0-1cceb6f0b634-node-exporter-accelerators-collector-config\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:05.157803 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.157721 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5522ff12-fed1-4411-96b0-1cceb6f0b634-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:05.157803 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.157746 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/95638663-fffb-41a6-8056-c1f4293a3fb0-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-8wz25\" (UID: \"95638663-fffb-41a6-8056-c1f4293a3fb0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" Nov 04 00:58:05.157803 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.157771 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/95638663-fffb-41a6-8056-c1f4293a3fb0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-8wz25\" (UID: \"95638663-fffb-41a6-8056-c1f4293a3fb0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" Nov 04 00:58:05.157803 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.157798 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-dl2dc\" (UID: \"1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dl2dc" Nov 04 00:58:05.158033 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.157832 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95638663-fffb-41a6-8056-c1f4293a3fb0-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-8wz25\" (UID: \"95638663-fffb-41a6-8056-c1f4293a3fb0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" Nov 04 00:58:05.158033 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.157853 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-dl2dc\" (UID: \"1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dl2dc" Nov 04 00:58:05.158033 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.157887 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5522ff12-fed1-4411-96b0-1cceb6f0b634-sys\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:05.158033 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.157965 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5522ff12-fed1-4411-96b0-1cceb6f0b634-node-exporter-textfile\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:05.158033 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.157988 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5522ff12-fed1-4411-96b0-1cceb6f0b634-node-exporter-tls\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:05.158033 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.158009 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5522ff12-fed1-4411-96b0-1cceb6f0b634-metrics-client-ca\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:05.158033 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.158034 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5522ff12-fed1-4411-96b0-1cceb6f0b634-root\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:05.158363 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.158053 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59m4r\" (UniqueName: \"kubernetes.io/projected/5522ff12-fed1-4411-96b0-1cceb6f0b634-kube-api-access-59m4r\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:05.158363 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.158074 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqzsj\" (UniqueName: \"kubernetes.io/projected/95638663-fffb-41a6-8056-c1f4293a3fb0-kube-api-access-hqzsj\") pod \"kube-state-metrics-5dc8d7d89b-8wz25\" (UID: \"95638663-fffb-41a6-8056-c1f4293a3fb0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" Nov 04 00:58:05.158363 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.158094 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/95638663-fffb-41a6-8056-c1f4293a3fb0-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-8wz25\" (UID: \"95638663-fffb-41a6-8056-c1f4293a3fb0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" Nov 04 00:58:05.158363 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.158121 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr5ms\" (UniqueName: \"kubernetes.io/projected/1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84-kube-api-access-wr5ms\") pod \"openshift-state-metrics-7bd7489957-dl2dc\" (UID: \"1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dl2dc" Nov 04 00:58:05.158363 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.158159 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95638663-fffb-41a6-8056-c1f4293a3fb0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-8wz25\" (UID: \"95638663-fffb-41a6-8056-c1f4293a3fb0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" Nov 04 00:58:05.158363 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.158184 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5522ff12-fed1-4411-96b0-1cceb6f0b634-node-exporter-wtmp\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:05.158363 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.158210 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-dl2dc\" (UID: \"1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dl2dc" Nov 04 00:58:05.158753 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.158408 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5522ff12-fed1-4411-96b0-1cceb6f0b634-node-exporter-accelerators-collector-config\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:05.158753 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.158619 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/95638663-fffb-41a6-8056-c1f4293a3fb0-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-8wz25\" (UID: \"95638663-fffb-41a6-8056-c1f4293a3fb0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" Nov 04 00:58:05.158753 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.158687 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5522ff12-fed1-4411-96b0-1cceb6f0b634-root\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:05.158901 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.158757 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5522ff12-fed1-4411-96b0-1cceb6f0b634-metrics-client-ca\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:05.158901 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.158853 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5522ff12-fed1-4411-96b0-1cceb6f0b634-node-exporter-wtmp\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:05.160212 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.159274 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5522ff12-fed1-4411-96b0-1cceb6f0b634-sys\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:05.160212 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.159646 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5522ff12-fed1-4411-96b0-1cceb6f0b634-node-exporter-textfile\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:05.160212 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.159788 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-dl2dc\" (UID: \"1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dl2dc" Nov 04 00:58:05.160212 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.159965 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/95638663-fffb-41a6-8056-c1f4293a3fb0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-8wz25\" (UID: \"95638663-fffb-41a6-8056-c1f4293a3fb0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" Nov 04 00:58:05.160212 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.160108 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95638663-fffb-41a6-8056-c1f4293a3fb0-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-8wz25\" (UID: \"95638663-fffb-41a6-8056-c1f4293a3fb0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" Nov 04 00:58:05.161845 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.161690 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5522ff12-fed1-4411-96b0-1cceb6f0b634-node-exporter-tls\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:05.161845 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.161789 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-dl2dc\" (UID: \"1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dl2dc" Nov 04 00:58:05.161845 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.161819 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/95638663-fffb-41a6-8056-c1f4293a3fb0-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-8wz25\" (UID: \"95638663-fffb-41a6-8056-c1f4293a3fb0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" Nov 04 00:58:05.162017 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.161875 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95638663-fffb-41a6-8056-c1f4293a3fb0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-8wz25\" (UID: \"95638663-fffb-41a6-8056-c1f4293a3fb0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" Nov 04 00:58:05.162017 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.161947 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5522ff12-fed1-4411-96b0-1cceb6f0b634-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:05.162158 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.162142 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-dl2dc\" (UID: \"1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dl2dc" Nov 04 00:58:05.168710 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.168686 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59m4r\" (UniqueName: \"kubernetes.io/projected/5522ff12-fed1-4411-96b0-1cceb6f0b634-kube-api-access-59m4r\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:05.169030 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.169008 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr5ms\" (UniqueName: \"kubernetes.io/projected/1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84-kube-api-access-wr5ms\") pod \"openshift-state-metrics-7bd7489957-dl2dc\" (UID: \"1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dl2dc" Nov 04 00:58:05.171323 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.171302 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqzsj\" (UniqueName: \"kubernetes.io/projected/95638663-fffb-41a6-8056-c1f4293a3fb0-kube-api-access-hqzsj\") pod \"kube-state-metrics-5dc8d7d89b-8wz25\" (UID: \"95638663-fffb-41a6-8056-c1f4293a3fb0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" Nov 04 00:58:05.256993 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.256897 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dl2dc" Nov 04 00:58:05.309367 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.309336 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" Nov 04 00:58:05.349453 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.349398 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:05.385736 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.385706 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:05.388521 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:58:05.388371 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5522ff12_fed1_4411_96b0_1cceb6f0b634.slice/crio-48c537a2e48500aa870420db10b2f9fe6ca3063600d111ce829f3d8342d9f429 WatchSource:0}: Error finding container 48c537a2e48500aa870420db10b2f9fe6ca3063600d111ce829f3d8342d9f429: Status 404 returned error can't find the container with id 48c537a2e48500aa870420db10b2f9fe6ca3063600d111ce829f3d8342d9f429 Nov 04 00:58:05.556091 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.556013 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-dl2dc"] Nov 04 00:58:05.559381 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:58:05.559354 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fdfa1d2_7870_4cf2_aaf3_ea7a919bfb84.slice/crio-350744c898150214cc41dce1779dab592400db1175c630c75b0851a35af5f404 WatchSource:0}: Error finding container 350744c898150214cc41dce1779dab592400db1175c630c75b0851a35af5f404: Status 404 returned error can't find the container with id 350744c898150214cc41dce1779dab592400db1175c630c75b0851a35af5f404 Nov 04 00:58:05.577187 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.577117 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" limit=1024 Nov 04 00:58:05.632407 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.632379 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25"] Nov 04 00:58:05.636017 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:58:05.635984 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95638663_fffb_41a6_8056_c1f4293a3fb0.slice/crio-ba5ce77d099cc1f6928cc92045a63bda220bc2e2fbd0568b148347b236ca589b WatchSource:0}: Error finding container ba5ce77d099cc1f6928cc92045a63bda220bc2e2fbd0568b148347b236ca589b: Status 404 returned error can't find the container with id ba5ce77d099cc1f6928cc92045a63bda220bc2e2fbd0568b148347b236ca589b Nov 04 00:58:05.939103 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.938845 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 04 00:58:05.954569 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.954539 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:05.977360 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.977336 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 04 00:58:05.977468 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.977433 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 04 00:58:05.977617 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.977602 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 04 00:58:05.977679 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.977650 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 04 00:58:05.977769 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.977748 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 04 00:58:05.977898 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.977796 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 04 00:58:05.977898 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.977815 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 04 00:58:05.977898 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.977826 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 04 00:58:05.977898 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.977853 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 04 00:58:05.977898 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.977880 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-llw9w\"" Nov 04 00:58:05.988625 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:05.988602 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 04 00:58:06.040203 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.040174 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mhb4l" event={"ID":"5522ff12-fed1-4411-96b0-1cceb6f0b634","Type":"ContainerStarted","Data":"48c537a2e48500aa870420db10b2f9fe6ca3063600d111ce829f3d8342d9f429"} Nov 04 00:58:06.065418 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.065395 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sprpf\" (UniqueName: \"kubernetes.io/projected/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-kube-api-access-sprpf\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.065547 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.065429 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.065547 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.065477 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.065646 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.065556 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.065646 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.065585 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.065646 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.065622 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.066143 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.066110 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-config-out\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.066225 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.066168 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-web-config\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.066374 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.066356 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.066451 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.066427 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.066504 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.066461 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.066504 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.066492 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-config-volume\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.066602 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.066542 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.071335 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.071312 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dl2dc" event={"ID":"1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84","Type":"ContainerStarted","Data":"b17070afabb0f93981958fe6a98a1e33acf1da9251cb5e266d45e2835bfdfe7f"} Nov 04 00:58:06.071542 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.071518 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dl2dc" event={"ID":"1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84","Type":"ContainerStarted","Data":"c8897e03175b964a5377cb944a2afac0126461e2ef6a37fc08afcc67bffd004f"} Nov 04 00:58:06.071641 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.071567 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dl2dc" event={"ID":"1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84","Type":"ContainerStarted","Data":"350744c898150214cc41dce1779dab592400db1175c630c75b0851a35af5f404"} Nov 04 00:58:06.072639 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.072602 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" event={"ID":"95638663-fffb-41a6-8056-c1f4293a3fb0","Type":"ContainerStarted","Data":"ba5ce77d099cc1f6928cc92045a63bda220bc2e2fbd0568b148347b236ca589b"} Nov 04 00:58:06.167152 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.167123 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.167271 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.167174 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.167271 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.167202 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-config-volume\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.167271 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.167252 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.167419 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.167281 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sprpf\" (UniqueName: \"kubernetes.io/projected/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-kube-api-access-sprpf\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.167419 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.167314 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.167419 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.167357 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.167553 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.167444 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.167553 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.167494 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.167553 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.167528 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.167696 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.167556 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-config-out\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.167696 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.167587 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-web-config\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.167696 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.167635 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.167999 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.167971 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.169502 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.168280 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.169502 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.169042 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.171567 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.171545 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-config-volume\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.173006 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.171549 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.173103 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.171555 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.173103 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.172897 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.173213 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.173189 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.174407 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.174361 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-web-config\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.175364 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.175342 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.177842 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.177819 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sprpf\" (UniqueName: \"kubernetes.io/projected/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-kube-api-access-sprpf\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.178208 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.178185 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-config-out\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.187782 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.187762 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.386095 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.386055 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:06.681033 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.680993 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 04 00:58:06.684504 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:58:06.684467 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf45dd13e_3bfc_49ee_80d8_8252ec2f352c.slice/crio-a8defe25cc00c430a9110887fea8d6dff2d0a65dfdb922885b8fdf364c92ae62 WatchSource:0}: Error finding container a8defe25cc00c430a9110887fea8d6dff2d0a65dfdb922885b8fdf364c92ae62: Status 404 returned error can't find the container with id a8defe25cc00c430a9110887fea8d6dff2d0a65dfdb922885b8fdf364c92ae62 Nov 04 00:58:06.923519 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.923484 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg"] Nov 04 00:58:06.926598 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.926021 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:06.932177 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.932145 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 04 00:58:06.932275 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.932200 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 04 00:58:06.932275 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.932250 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 04 00:58:06.932275 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.932198 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 04 00:58:06.932434 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.932293 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 04 00:58:06.932559 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.932538 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-f8bj3puo34rtq\"" Nov 04 00:58:06.932636 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.932602 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-rpsh2\"" Nov 04 00:58:06.942730 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:06.942710 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg"] Nov 04 00:58:07.074509 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.074467 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-metrics-client-ca\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:07.074509 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.074502 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-thanos-querier-tls\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:07.075374 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.074629 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:07.075374 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.074665 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-tmp\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:07.075374 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.074750 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prd7v\" (UniqueName: \"kubernetes.io/projected/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-kube-api-access-prd7v\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:07.075374 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.074783 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-grpc-tls\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:07.075374 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.074815 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:07.075374 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.074943 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:07.075374 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.075008 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:07.079412 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.079307 2577 generic.go:356] "Generic (PLEG): container finished" podID="5522ff12-fed1-4411-96b0-1cceb6f0b634" containerID="5cada670732f2ccab326c7baf7642add2f207334e3b96ee66f50a2f68552a493" exitCode=0 Nov 04 00:58:07.079549 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.079403 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mhb4l" event={"ID":"5522ff12-fed1-4411-96b0-1cceb6f0b634","Type":"ContainerDied","Data":"5cada670732f2ccab326c7baf7642add2f207334e3b96ee66f50a2f68552a493"} Nov 04 00:58:07.081299 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.081225 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f45dd13e-3bfc-49ee-80d8-8252ec2f352c","Type":"ContainerStarted","Data":"a8defe25cc00c430a9110887fea8d6dff2d0a65dfdb922885b8fdf364c92ae62"} Nov 04 00:58:07.083695 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.083661 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cv4w2" event={"ID":"d90c5697-ccdd-4845-8972-dbbd952dcc87","Type":"ContainerStarted","Data":"1c3b0606e5c1d38ae11dc04e6d34ad22c3bf2775a56ce8eae487a4237445f03f"} Nov 04 00:58:07.088549 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.088530 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7jm5h" event={"ID":"59e9f4f9-590f-43b0-999d-1fad31779d16","Type":"ContainerStarted","Data":"5a98f907dc3f969e97021cb91b23c4c92dc696df473ec46f31ca05f77c98e35f"} Nov 04 00:58:07.140329 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.140274 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7jm5h" podStartSLOduration=2.35391522 podStartE2EDuration="6.14025608s" totalImagesPullingTime="3.78634086s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:58:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:07.139449218 +0000 UTC m=+44.051615735" watchObservedRunningTime="2026-11-04 00:58:07.14025608 +0000 UTC m=+44.052422598" Nov 04 00:58:07.156797 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.156758 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cv4w2" podStartSLOduration=2.062920879 podStartE2EDuration="6.156732282s" totalImagesPullingTime="4.093811403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:58:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:07.15578312 +0000 UTC m=+44.067949636" watchObservedRunningTime="2026-11-04 00:58:07.156732282 +0000 UTC m=+44.068898799" Nov 04 00:58:07.176679 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.176647 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-thanos-querier-tls\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:07.176825 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.176807 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:07.176897 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.176882 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-tmp\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:07.176984 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.176957 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prd7v\" (UniqueName: \"kubernetes.io/projected/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-kube-api-access-prd7v\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:07.177041 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.177016 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-grpc-tls\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:07.177101 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.177085 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:07.177165 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.177153 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:07.177221 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.177208 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:07.177274 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.177257 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-metrics-client-ca\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:07.180662 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.180632 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-metrics-client-ca\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:07.181235 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.181203 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-tmp\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:07.181706 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.181685 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-thanos-querier-tls\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:07.186411 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.186375 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-grpc-tls\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:07.187031 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.186499 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:07.187031 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.186981 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:07.187766 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.187697 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:07.188170 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.188147 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:07.192480 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.192458 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prd7v\" (UniqueName: \"kubernetes.io/projected/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-kube-api-access-prd7v\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:07.252146 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.251988 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:07.543351 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:07.543316 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg"] Nov 04 00:58:07.548256 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:58:07.548221 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c5b0800_f9a2_4ce6_a3d8_5459cbf9a70a.slice/crio-ff0bc0fe6e6675667a340512c877189dc978ad51e714f5e355f4cfa516eb9eae WatchSource:0}: Error finding container ff0bc0fe6e6675667a340512c877189dc978ad51e714f5e355f4cfa516eb9eae: Status 404 returned error can't find the container with id ff0bc0fe6e6675667a340512c877189dc978ad51e714f5e355f4cfa516eb9eae Nov 04 00:58:08.093578 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:08.093500 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" event={"ID":"95638663-fffb-41a6-8056-c1f4293a3fb0","Type":"ContainerStarted","Data":"418e28f9766298f733b8a6a5fc9b05e67fca664f210c29b3bef9715985eb74f0"} Nov 04 00:58:08.094299 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:08.093575 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" event={"ID":"95638663-fffb-41a6-8056-c1f4293a3fb0","Type":"ContainerStarted","Data":"f2776e9eea5c274aba52e3427b5f1de3ba3a4a89d13e84777b57906560d1be2c"} Nov 04 00:58:08.094299 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:08.093606 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" event={"ID":"95638663-fffb-41a6-8056-c1f4293a3fb0","Type":"ContainerStarted","Data":"06649c3d9e9ba0773247f6ff2f30d09ce8e974c4a5ea55a57fe3308e10927dc9"} Nov 04 00:58:08.095729 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:08.095706 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mhb4l" event={"ID":"5522ff12-fed1-4411-96b0-1cceb6f0b634","Type":"ContainerStarted","Data":"aed90e2708ae974414db46d417187fc13f5d7349eb1f167807808f41c9b3c87d"} Nov 04 00:58:08.095833 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:08.095745 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mhb4l" event={"ID":"5522ff12-fed1-4411-96b0-1cceb6f0b634","Type":"ContainerStarted","Data":"bfa2360206f62737d98f3883737ea596906c6919447aad1c6292ddb04d9977f1"} Nov 04 00:58:08.097368 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:08.097340 2577 generic.go:356] "Generic (PLEG): container finished" podID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerID="223d2944c74314599510a4f7c0349335629caf4e673f355fc2d608fd232009d7" exitCode=0 Nov 04 00:58:08.097443 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:08.097388 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f45dd13e-3bfc-49ee-80d8-8252ec2f352c","Type":"ContainerDied","Data":"223d2944c74314599510a4f7c0349335629caf4e673f355fc2d608fd232009d7"} Nov 04 00:58:08.099711 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:08.099692 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dl2dc" event={"ID":"1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84","Type":"ContainerStarted","Data":"82f721f3f84a2869c32b2642ed090bdad412151f39ad11f7d5144eafb81399cc"} Nov 04 00:58:08.101150 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:08.101129 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" event={"ID":"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a","Type":"ContainerStarted","Data":"ff0bc0fe6e6675667a340512c877189dc978ad51e714f5e355f4cfa516eb9eae"} Nov 04 00:58:08.116028 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:08.115989 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" podStartSLOduration=2.143447828 podStartE2EDuration="4.115977658s" totalImagesPullingTime="1.97252983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:08.115083009 +0000 UTC m=+45.027249529" watchObservedRunningTime="2026-11-04 00:58:08.115977658 +0000 UTC m=+45.028144174" Nov 04 00:58:08.171348 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:08.171305 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dl2dc" podStartSLOduration=2.78765568 podStartE2EDuration="4.171290184s" totalImagesPullingTime="1.383634504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:08.171173695 +0000 UTC m=+45.083340213" watchObservedRunningTime="2026-11-04 00:58:08.171290184 +0000 UTC m=+45.083456700" Nov 04 00:58:08.196367 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:08.196330 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mhb4l" podStartSLOduration=2.6433605829999998 podStartE2EDuration="4.196319098s" totalImagesPullingTime="1.450711092s" totalInitContainerRuntime="102.247423ms" isStatefulPod=true podCreationTimestamp="2026-11-04 00:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:08.195134608 +0000 UTC m=+45.107301126" watchObservedRunningTime="2026-11-04 00:58:08.196319098 +0000 UTC m=+45.108485614" Nov 04 00:58:09.378286 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.378235 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-845fbdd675-9rvps"] Nov 04 00:58:09.382456 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.381266 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:09.390441 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.390191 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 04 00:58:09.390441 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.390293 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 04 00:58:09.392543 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.390768 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 04 00:58:09.392543 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.390844 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-en40opnp7to2v\"" Nov 04 00:58:09.392543 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.390888 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-97qhl\"" Nov 04 00:58:09.392543 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.390965 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-845fbdd675-9rvps"] Nov 04 00:58:09.392543 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.390988 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 04 00:58:09.501729 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.501695 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/559e1de2-7e21-4aeb-916f-7a75e0c75841-audit-log\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:09.501906 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.501745 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/559e1de2-7e21-4aeb-916f-7a75e0c75841-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:09.501906 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.501830 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/559e1de2-7e21-4aeb-916f-7a75e0c75841-metrics-server-audit-profiles\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:09.501906 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.501877 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/559e1de2-7e21-4aeb-916f-7a75e0c75841-secret-metrics-server-client-certs\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:09.501906 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.501899 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/559e1de2-7e21-4aeb-916f-7a75e0c75841-client-ca-bundle\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:09.502130 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.501949 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfpdd\" (UniqueName: \"kubernetes.io/projected/559e1de2-7e21-4aeb-916f-7a75e0c75841-kube-api-access-bfpdd\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:09.502130 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.501978 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/559e1de2-7e21-4aeb-916f-7a75e0c75841-secret-metrics-server-tls\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:09.602872 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.602836 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/559e1de2-7e21-4aeb-916f-7a75e0c75841-audit-log\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:09.603047 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.602899 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/559e1de2-7e21-4aeb-916f-7a75e0c75841-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:09.603047 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.602959 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/559e1de2-7e21-4aeb-916f-7a75e0c75841-metrics-server-audit-profiles\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:09.603047 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.603006 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/559e1de2-7e21-4aeb-916f-7a75e0c75841-secret-metrics-server-client-certs\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:09.603047 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.603035 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/559e1de2-7e21-4aeb-916f-7a75e0c75841-client-ca-bundle\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:09.603173 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.603072 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfpdd\" (UniqueName: \"kubernetes.io/projected/559e1de2-7e21-4aeb-916f-7a75e0c75841-kube-api-access-bfpdd\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:09.603173 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.603101 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/559e1de2-7e21-4aeb-916f-7a75e0c75841-secret-metrics-server-tls\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:09.604194 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.603961 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/559e1de2-7e21-4aeb-916f-7a75e0c75841-audit-log\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:09.604587 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.604541 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/559e1de2-7e21-4aeb-916f-7a75e0c75841-metrics-server-audit-profiles\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:09.605168 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.605136 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/559e1de2-7e21-4aeb-916f-7a75e0c75841-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:09.606635 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.606609 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/559e1de2-7e21-4aeb-916f-7a75e0c75841-secret-metrics-server-tls\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:09.607547 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.607167 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/559e1de2-7e21-4aeb-916f-7a75e0c75841-secret-metrics-server-client-certs\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:09.607547 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.607504 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/559e1de2-7e21-4aeb-916f-7a75e0c75841-client-ca-bundle\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:09.613269 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.613243 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfpdd\" (UniqueName: \"kubernetes.io/projected/559e1de2-7e21-4aeb-916f-7a75e0c75841-kube-api-access-bfpdd\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:09.691858 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.691829 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-4z274"] Nov 04 00:58:09.693836 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.693808 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4z274" Nov 04 00:58:09.696149 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.696128 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:09.697601 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.697580 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 04 00:58:09.697735 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.697715 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-qw5ml\"" Nov 04 00:58:09.704114 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.704001 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-4z274"] Nov 04 00:58:09.805255 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.805230 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5bcf3164-e6e8-4323-aad0-ee8adbe4a77d-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4z274\" (UID: \"5bcf3164-e6e8-4323-aad0-ee8adbe4a77d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4z274" Nov 04 00:58:09.805366 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.805271 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5ssp\" (UniqueName: \"kubernetes.io/projected/5bcf3164-e6e8-4323-aad0-ee8adbe4a77d-kube-api-access-n5ssp\") pod \"monitoring-plugin-77f4f76594-4z274\" (UID: \"5bcf3164-e6e8-4323-aad0-ee8adbe4a77d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4z274" Nov 04 00:58:09.907076 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.906734 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5bcf3164-e6e8-4323-aad0-ee8adbe4a77d-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4z274\" (UID: \"5bcf3164-e6e8-4323-aad0-ee8adbe4a77d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4z274" Nov 04 00:58:09.907076 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.906780 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5ssp\" (UniqueName: \"kubernetes.io/projected/5bcf3164-e6e8-4323-aad0-ee8adbe4a77d-kube-api-access-n5ssp\") pod \"monitoring-plugin-77f4f76594-4z274\" (UID: \"5bcf3164-e6e8-4323-aad0-ee8adbe4a77d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4z274" Nov 04 00:58:09.909175 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.909155 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5bcf3164-e6e8-4323-aad0-ee8adbe4a77d-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4z274\" (UID: \"5bcf3164-e6e8-4323-aad0-ee8adbe4a77d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4z274" Nov 04 00:58:09.915284 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.915262 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5ssp\" (UniqueName: \"kubernetes.io/projected/5bcf3164-e6e8-4323-aad0-ee8adbe4a77d-kube-api-access-n5ssp\") pod \"monitoring-plugin-77f4f76594-4z274\" (UID: \"5bcf3164-e6e8-4323-aad0-ee8adbe4a77d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4z274" Nov 04 00:58:09.985241 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:09.985213 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-845fbdd675-9rvps"] Nov 04 00:58:10.030850 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.030489 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4z274" Nov 04 00:58:10.111095 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.111071 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" event={"ID":"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a","Type":"ContainerStarted","Data":"e143f7051b33da7cec85a81d9180640fe0560a91099c6d22884da270ac3313a3"} Nov 04 00:58:10.111165 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.111133 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" event={"ID":"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a","Type":"ContainerStarted","Data":"ba2c7bc7fa2a97e5ab2b5372e81e6d94d8121d071ba21af7eabed5fc66a0678f"} Nov 04 00:58:10.114465 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.114303 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f45dd13e-3bfc-49ee-80d8-8252ec2f352c","Type":"ContainerStarted","Data":"e5c8d787a70a6a5bb39ecdfa4b7346a21df9ed1b5e7546f30e0fef7669aaac9e"} Nov 04 00:58:10.114465 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.114352 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f45dd13e-3bfc-49ee-80d8-8252ec2f352c","Type":"ContainerStarted","Data":"adf1116db30959de5b988e09caba6de90c86c3d8eac0b536bc15e7d031ab3035"} Nov 04 00:58:10.114465 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.114376 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f45dd13e-3bfc-49ee-80d8-8252ec2f352c","Type":"ContainerStarted","Data":"444d87bd8b809346877521d4160630c88f94461cd7f5ceff86a439d468197199"} Nov 04 00:58:10.114465 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.114401 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f45dd13e-3bfc-49ee-80d8-8252ec2f352c","Type":"ContainerStarted","Data":"78cfe9bd056d53b1760fe77c2b25381fda1cf7e8e227c08c676986daffd8c2af"} Nov 04 00:58:10.114465 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.114422 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f45dd13e-3bfc-49ee-80d8-8252ec2f352c","Type":"ContainerStarted","Data":"b461d5ec652164a2611aa3a9967068acd7e76e9cbca9e03152786644e970d82b"} Nov 04 00:58:10.115617 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.115597 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" event={"ID":"559e1de2-7e21-4aeb-916f-7a75e0c75841","Type":"ContainerStarted","Data":"84fecc8827c4ab0bb51183624b9a312af52113fa78b2052b8e875bf8289eb44a"} Nov 04 00:58:10.268728 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.268660 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-69dd78b66f-clp8w"] Nov 04 00:58:10.270706 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.270692 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:10.275659 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.275638 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 04 00:58:10.276054 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.275980 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 04 00:58:10.276054 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.276030 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 04 00:58:10.276222 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.276062 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 04 00:58:10.276222 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.276167 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-q5sdf\"" Nov 04 00:58:10.276463 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.276446 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 04 00:58:10.276539 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.276442 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 04 00:58:10.276539 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.276469 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 04 00:58:10.285624 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.285599 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69dd78b66f-clp8w"] Nov 04 00:58:10.285698 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.285679 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 04 00:58:10.318303 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.318280 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-4z274"] Nov 04 00:58:10.320883 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:58:10.320854 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bcf3164_e6e8_4323_aad0_ee8adbe4a77d.slice/crio-0ee3f067670f8fcf04bafdaf8967f414cf98590768a1670fa6221c71f63157d6 WatchSource:0}: Error finding container 0ee3f067670f8fcf04bafdaf8967f414cf98590768a1670fa6221c71f63157d6: Status 404 returned error can't find the container with id 0ee3f067670f8fcf04bafdaf8967f414cf98590768a1670fa6221c71f63157d6 Nov 04 00:58:10.410444 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.410420 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/34aecbea-ff97-4f23-8336-326316720a2f-console-oauth-config\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:10.410731 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.410460 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-console-config\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:10.410731 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.410487 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-trusted-ca-bundle\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:10.410731 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.410600 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79nvt\" (UniqueName: \"kubernetes.io/projected/34aecbea-ff97-4f23-8336-326316720a2f-kube-api-access-79nvt\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:10.410731 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.410646 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/34aecbea-ff97-4f23-8336-326316720a2f-console-serving-cert\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:10.410731 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.410665 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-service-ca\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:10.410731 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.410690 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34aecbea-ff97-4f23-8336-326316720a2f-tmp\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:10.410913 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.410783 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-oauth-serving-cert\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:10.511725 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.511702 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-oauth-serving-cert\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:10.511804 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.511745 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/34aecbea-ff97-4f23-8336-326316720a2f-console-oauth-config\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:10.511804 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.511769 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-console-config\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:10.511804 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.511784 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-trusted-ca-bundle\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:10.511976 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.511955 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79nvt\" (UniqueName: \"kubernetes.io/projected/34aecbea-ff97-4f23-8336-326316720a2f-kube-api-access-79nvt\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:10.512030 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.512010 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/34aecbea-ff97-4f23-8336-326316720a2f-console-serving-cert\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:10.512091 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.512039 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-service-ca\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:10.512091 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.512079 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34aecbea-ff97-4f23-8336-326316720a2f-tmp\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:10.512463 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.512435 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34aecbea-ff97-4f23-8336-326316720a2f-tmp\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:10.512547 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.512481 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-console-config\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:10.512668 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.512647 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-trusted-ca-bundle\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:10.512808 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.512788 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-service-ca\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:10.512855 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.512828 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-oauth-serving-cert\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:10.514271 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.514250 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/34aecbea-ff97-4f23-8336-326316720a2f-console-oauth-config\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:10.514439 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.514419 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/34aecbea-ff97-4f23-8336-326316720a2f-console-serving-cert\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:10.520130 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.520084 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79nvt\" (UniqueName: \"kubernetes.io/projected/34aecbea-ff97-4f23-8336-326316720a2f-kube-api-access-79nvt\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:10.583629 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.583604 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:10.857969 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:10.857866 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69dd78b66f-clp8w"] Nov 04 00:58:10.864615 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:58:10.864585 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34aecbea_ff97_4f23_8336_326316720a2f.slice/crio-542986989c9fe1332f4a001b80ae9f7c3a566c325acb0350072842372c21b519 WatchSource:0}: Error finding container 542986989c9fe1332f4a001b80ae9f7c3a566c325acb0350072842372c21b519: Status 404 returned error can't find the container with id 542986989c9fe1332f4a001b80ae9f7c3a566c325acb0350072842372c21b519 Nov 04 00:58:11.120843 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.120812 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69dd78b66f-clp8w" event={"ID":"34aecbea-ff97-4f23-8336-326316720a2f","Type":"ContainerStarted","Data":"542986989c9fe1332f4a001b80ae9f7c3a566c325acb0350072842372c21b519"} Nov 04 00:58:11.122366 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.122337 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4z274" event={"ID":"5bcf3164-e6e8-4323-aad0-ee8adbe4a77d","Type":"ContainerStarted","Data":"0ee3f067670f8fcf04bafdaf8967f414cf98590768a1670fa6221c71f63157d6"} Nov 04 00:58:11.124790 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.124764 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" event={"ID":"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a","Type":"ContainerStarted","Data":"f5bd402f72b1096839ab8c3c5e0c520d540410b980b8cb862d67cc2751162df1"} Nov 04 00:58:11.240861 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.240749 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 00:58:11.244262 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.244241 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.260046 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.260012 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 04 00:58:11.260254 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.260239 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 04 00:58:11.260413 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.260399 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 04 00:58:11.269998 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.269692 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 04 00:58:11.271369 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.271336 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 04 00:58:11.271369 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.271365 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 04 00:58:11.275021 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.275000 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 00:58:11.275186 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.275168 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 04 00:58:11.275253 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.275224 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 04 00:58:11.275289 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.275255 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 04 00:58:11.360948 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.360867 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 04 00:58:11.361235 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.361217 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 04 00:58:11.361582 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.361420 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9outkgf7grlms\"" Nov 04 00:58:11.361582 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.361454 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 04 00:58:11.361582 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.361490 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 04 00:58:11.361582 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.361532 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wzbxc\"" Nov 04 00:58:11.361582 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.361563 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 04 00:58:11.420448 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.420374 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.420448 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.420432 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.420951 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.420540 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.420951 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.420601 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.420951 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.420648 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.420951 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.420718 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.420951 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.420764 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.420951 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.420787 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95jzn\" (UniqueName: \"kubernetes.io/projected/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-kube-api-access-95jzn\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.420951 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.420821 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.421292 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.420957 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-config\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.421292 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.420985 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.421292 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.421013 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.421292 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.421039 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-web-config\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.421292 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.421070 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.421292 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.421097 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.421292 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.421144 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.421292 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.421172 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.421292 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.421209 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.421292 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.421236 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-config-out\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.421292 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.421258 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.522704 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.522664 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.522864 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.522728 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-config\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.522864 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.522755 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.523016 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.522910 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.529696 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.529669 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-web-config\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.529809 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.529720 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.529809 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.529756 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.529913 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.529811 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.529913 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.529842 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.529913 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.529873 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.529913 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.529900 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-config-out\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.530075 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.529942 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.530075 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.529979 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.530075 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.530030 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.530075 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.530057 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.530212 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.530099 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.530212 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.530125 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.530212 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.530155 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.530212 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.530201 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.530354 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.530233 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95jzn\" (UniqueName: \"kubernetes.io/projected/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-kube-api-access-95jzn\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.531061 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.531016 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.531341 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.531319 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.532670 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.531516 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.532670 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.531869 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.532670 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.532088 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.534333 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.534303 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.536456 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.535167 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.536456 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.535840 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.536456 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.536045 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.537676 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.537350 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.538987 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.538962 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.540449 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.540429 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.542655 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.542633 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.542748 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.542629 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.542748 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.542708 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-config\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.543008 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.542987 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-web-config\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.543912 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.543353 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.544190 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.544163 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-config-out\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.544988 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.544889 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.544988 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.544952 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95jzn\" (UniqueName: \"kubernetes.io/projected/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-kube-api-access-95jzn\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.575724 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.575699 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:11.878182 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:11.878149 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 00:58:11.883000 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:58:11.882964 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0ce8d6_9cfb_4249_96ca_3866749af7f7.slice/crio-2d2525aeb17bd4389865536271eac885aad37b39e20e47e47da5fbc206bb7099 WatchSource:0}: Error finding container 2d2525aeb17bd4389865536271eac885aad37b39e20e47e47da5fbc206bb7099: Status 404 returned error can't find the container with id 2d2525aeb17bd4389865536271eac885aad37b39e20e47e47da5fbc206bb7099 Nov 04 00:58:12.134535 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:12.134459 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f45dd13e-3bfc-49ee-80d8-8252ec2f352c","Type":"ContainerStarted","Data":"6effbe28eed4891cb772305777544e80ea635a5a35153f8cfa55897562a48642"} Nov 04 00:58:12.136717 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:12.136689 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" event={"ID":"559e1de2-7e21-4aeb-916f-7a75e0c75841","Type":"ContainerStarted","Data":"d158cb67a22a049d51b8bce30e561ef53d980336e2985eb3c3ad64e15a9a240b"} Nov 04 00:58:12.140678 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:12.140651 2577 generic.go:356] "Generic (PLEG): container finished" podID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerID="1449e5c536ddfc17d9303f1dc08eb2d596ecca01af40809b528ae00d0b6f96e1" exitCode=0 Nov 04 00:58:12.140827 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:12.140732 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad0ce8d6-9cfb-4249-96ca-3866749af7f7","Type":"ContainerDied","Data":"1449e5c536ddfc17d9303f1dc08eb2d596ecca01af40809b528ae00d0b6f96e1"} Nov 04 00:58:12.140827 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:12.140806 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad0ce8d6-9cfb-4249-96ca-3866749af7f7","Type":"ContainerStarted","Data":"2d2525aeb17bd4389865536271eac885aad37b39e20e47e47da5fbc206bb7099"} Nov 04 00:58:12.145695 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:12.145670 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" event={"ID":"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a","Type":"ContainerStarted","Data":"7d76cd3ecdf9a58a6e84375d5a4c666db78a3c3088f0de9a418cf8cbd6089cdb"} Nov 04 00:58:12.145805 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:12.145737 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" event={"ID":"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a","Type":"ContainerStarted","Data":"fb4d05a2494be3201d67aadfe89bde9f63fa60c3500a70736363df7df80da5b1"} Nov 04 00:58:12.145805 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:12.145783 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" event={"ID":"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a","Type":"ContainerStarted","Data":"f8c5d8d002d30261a48e91415fc48679eeb0ddf96c45778f471d68c4a227684b"} Nov 04 00:58:12.145906 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:12.145832 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:12.162863 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:12.162806 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.21584252 podStartE2EDuration="7.162788749s" totalImagesPullingTime="3.933401189s" totalInitContainerRuntime="13.54504ms" isStatefulPod=false podCreationTimestamp="2026-11-04 00:58:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:12.162374712 +0000 UTC m=+49.074541242" watchObservedRunningTime="2026-11-04 00:58:12.162788749 +0000 UTC m=+49.074955267" Nov 04 00:58:12.209904 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:12.209841 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" podStartSLOduration=1.2822554720000001 podStartE2EDuration="3.209821386s" totalImagesPullingTime="1.927565914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:58:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:12.208903887 +0000 UTC m=+49.121070413" watchObservedRunningTime="2026-11-04 00:58:12.209821386 +0000 UTC m=+49.121987903" Nov 04 00:58:12.210976 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:12.210901 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" podStartSLOduration=2.803183893 podStartE2EDuration="6.210888562s" totalImagesPullingTime="3.407704669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:58:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:12.189750947 +0000 UTC m=+49.101917466" watchObservedRunningTime="2026-11-04 00:58:12.210888562 +0000 UTC m=+49.123055082" Nov 04 00:58:13.153745 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:13.153659 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4z274" event={"ID":"5bcf3164-e6e8-4323-aad0-ee8adbe4a77d","Type":"ContainerStarted","Data":"dd45e56544bd50ae3be5de96815b70520d0f7abe82e5f450e79b59d1d5655ac0"} Nov 04 00:58:13.176335 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:13.176280 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4z274" podStartSLOduration=1.817188451 podStartE2EDuration="4.17626702s" totalImagesPullingTime="2.359078569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:58:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:13.175595938 +0000 UTC m=+50.087762456" watchObservedRunningTime="2026-11-04 00:58:13.17626702 +0000 UTC m=+50.088433582" Nov 04 00:58:14.158285 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:14.158263 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4z274" Nov 04 00:58:14.164306 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:14.164282 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4z274" Nov 04 00:58:15.164626 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:15.164593 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69dd78b66f-clp8w" event={"ID":"34aecbea-ff97-4f23-8336-326316720a2f","Type":"ContainerStarted","Data":"a7efcc09b9005f34d05af2ff13cd52b5d3531141b639d5556415b2a0543ca983"} Nov 04 00:58:15.188874 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:15.188817 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-69dd78b66f-clp8w" podStartSLOduration=1.945753067 podStartE2EDuration="5.18880235s" totalImagesPullingTime="3.243049283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:58:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:15.186427919 +0000 UTC m=+52.098594438" watchObservedRunningTime="2026-11-04 00:58:15.18880235 +0000 UTC m=+52.100968888" Nov 04 00:58:16.091427 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:16.091306 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hhndm" Nov 04 00:58:16.173130 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:16.173108 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad0ce8d6-9cfb-4249-96ca-3866749af7f7","Type":"ContainerStarted","Data":"5d9a88f9c5dde43cfecfc6ea742cafb158898614f734f5010a0b9464fc63c044"} Nov 04 00:58:16.173519 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:16.173496 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad0ce8d6-9cfb-4249-96ca-3866749af7f7","Type":"ContainerStarted","Data":"afae6491578532a4bef9c8acd31e9b111e435252399947ae1f2fd00ca59089ca"} Nov 04 00:58:16.173605 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:16.173550 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad0ce8d6-9cfb-4249-96ca-3866749af7f7","Type":"ContainerStarted","Data":"080c05bf8c8df8fd6944be22f641c4387b191c82c18e2f3ae7ae692294c8c524"} Nov 04 00:58:16.173605 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:16.173594 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad0ce8d6-9cfb-4249-96ca-3866749af7f7","Type":"ContainerStarted","Data":"e906195f126edeb824eec62dc18fff594496021354341d88597607dba034b945"} Nov 04 00:58:16.173705 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:16.173635 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad0ce8d6-9cfb-4249-96ca-3866749af7f7","Type":"ContainerStarted","Data":"1c1a21c4d2257675634b3e4d9ef1aa95b4b9e7f748cf77988f3085f0a4b9bc14"} Nov 04 00:58:16.173705 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:16.173673 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad0ce8d6-9cfb-4249-96ca-3866749af7f7","Type":"ContainerStarted","Data":"ab68ecb00853ae770662fdd183ad9f573a55ee801404bdab3b6a5ece6b989e3d"} Nov 04 00:58:16.207954 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:16.207871 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.586190504 podStartE2EDuration="5.207851978s" totalImagesPullingTime="3.590668345s" totalInitContainerRuntime="30.993129ms" isStatefulPod=false podCreationTimestamp="2026-11-04 00:58:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:16.203757946 +0000 UTC m=+53.115924467" watchObservedRunningTime="2026-11-04 00:58:16.207851978 +0000 UTC m=+53.120018496" Nov 04 00:58:16.577526 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:16.577492 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:18.162061 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:18.162036 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:19.983386 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:19.983354 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:20.584489 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:20.584460 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:20.584648 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:20.584524 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:20.589155 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:20.589135 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:21.199288 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:21.199261 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:29.321437 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:29.321400 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bac70f0f-4585-42fe-abba-916ac41fcc62-original-pull-secret\") pod \"global-pull-secret-syncer-2x7kl\" (UID: \"bac70f0f-4585-42fe-abba-916ac41fcc62\") " pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:58:29.321918 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:29.321460 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d7sj\" (UniqueName: \"kubernetes.io/projected/9481376a-a63f-46e2-a605-fe4bb019c331-kube-api-access-6d7sj\") pod \"network-check-target-289kb\" (UID: \"9481376a-a63f-46e2-a605-fe4bb019c331\") " pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:58:29.321918 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:29.321515 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d232021-de4b-4240-8877-e99fe3563101-metrics-certs\") pod \"network-metrics-daemon-l4d28\" (UID: \"5d232021-de4b-4240-8877-e99fe3563101\") " pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:58:29.325988 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:29.325967 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 04 00:58:29.326189 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:29.326173 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 00:58:29.326340 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:29.326324 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 00:58:29.334943 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:29.334897 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d232021-de4b-4240-8877-e99fe3563101-metrics-certs\") pod \"network-metrics-daemon-l4d28\" (UID: \"5d232021-de4b-4240-8877-e99fe3563101\") " pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:58:29.335061 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:29.335046 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bac70f0f-4585-42fe-abba-916ac41fcc62-original-pull-secret\") pod \"global-pull-secret-syncer-2x7kl\" (UID: \"bac70f0f-4585-42fe-abba-916ac41fcc62\") " pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:58:29.335137 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:29.335121 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 04 00:58:29.345451 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:29.345431 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d7sj\" (UniqueName: \"kubernetes.io/projected/9481376a-a63f-46e2-a605-fe4bb019c331-kube-api-access-6d7sj\") pod \"network-check-target-289kb\" (UID: \"9481376a-a63f-46e2-a605-fe4bb019c331\") " pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:58:29.490775 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:29.490743 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-nszlw\"" Nov 04 00:58:29.497961 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:29.497944 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:58:29.506563 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:29.506545 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4rmxc\"" Nov 04 00:58:29.513864 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:29.513840 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:58:29.545837 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:29.545814 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q48lg\"" Nov 04 00:58:29.553648 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:29.553000 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:58:29.696568 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:29.696532 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:29.696696 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:29.696595 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:29.789668 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:29.789642 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2x7kl"] Nov 04 00:58:29.791796 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:58:29.791773 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac70f0f_4585_42fe_abba_916ac41fcc62.slice/crio-59fe24e8975ca0ad63c33c5e31b5d30e6cdf24e5497c94ca50e6c25c4015a842 WatchSource:0}: Error finding container 59fe24e8975ca0ad63c33c5e31b5d30e6cdf24e5497c94ca50e6c25c4015a842: Status 404 returned error can't find the container with id 59fe24e8975ca0ad63c33c5e31b5d30e6cdf24e5497c94ca50e6c25c4015a842 Nov 04 00:58:29.810197 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:29.810176 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-289kb"] Nov 04 00:58:29.811682 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:58:29.811662 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9481376a_a63f_46e2_a605_fe4bb019c331.slice/crio-2e471c8b707b2bbd6c778f9662b831117839d3f88b59ca306212a3a5091e8077 WatchSource:0}: Error finding container 2e471c8b707b2bbd6c778f9662b831117839d3f88b59ca306212a3a5091e8077: Status 404 returned error can't find the container with id 2e471c8b707b2bbd6c778f9662b831117839d3f88b59ca306212a3a5091e8077 Nov 04 00:58:29.847865 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:29.847846 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l4d28"] Nov 04 00:58:29.849574 ip-10-0-128-129 kubenswrapper[2577]: W1104 00:58:29.849553 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d232021_de4b_4240_8877_e99fe3563101.slice/crio-975451b57713ff9aaaaf05dc0a6904b669246c28463e3c2012c1bb33393b0376 WatchSource:0}: Error finding container 975451b57713ff9aaaaf05dc0a6904b669246c28463e3c2012c1bb33393b0376: Status 404 returned error can't find the container with id 975451b57713ff9aaaaf05dc0a6904b669246c28463e3c2012c1bb33393b0376 Nov 04 00:58:30.228052 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:30.228017 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l4d28" event={"ID":"5d232021-de4b-4240-8877-e99fe3563101","Type":"ContainerStarted","Data":"975451b57713ff9aaaaf05dc0a6904b669246c28463e3c2012c1bb33393b0376"} Nov 04 00:58:30.229258 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:30.229227 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-289kb" event={"ID":"9481376a-a63f-46e2-a605-fe4bb019c331","Type":"ContainerStarted","Data":"2e471c8b707b2bbd6c778f9662b831117839d3f88b59ca306212a3a5091e8077"} Nov 04 00:58:30.230256 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:30.230231 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2x7kl" event={"ID":"bac70f0f-4585-42fe-abba-916ac41fcc62","Type":"ContainerStarted","Data":"59fe24e8975ca0ad63c33c5e31b5d30e6cdf24e5497c94ca50e6c25c4015a842"} Nov 04 00:58:31.870521 ip-10-0-128-129 kubenswrapper[2577]: E1104 00:58:31.870423 2577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 00:58:32.241611 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:32.241573 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l4d28" event={"ID":"5d232021-de4b-4240-8877-e99fe3563101","Type":"ContainerStarted","Data":"40c5f4824ab84088f5cb73ca799448f006988eb078913f48060f87a48a9c0572"} Nov 04 00:58:32.241802 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:32.241642 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l4d28" event={"ID":"5d232021-de4b-4240-8877-e99fe3563101","Type":"ContainerStarted","Data":"e16448a8d0435992cdb0da8093b1b56c764c7c65e830e84c5bc446615b43267b"} Nov 04 00:58:32.263949 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:32.263866 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-l4d28" podStartSLOduration=67.965058976 podStartE2EDuration="1m9.263848442s" totalImagesPullingTime="1.298789466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:57:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:32.26091457 +0000 UTC m=+69.173081086" watchObservedRunningTime="2026-11-04 00:58:32.263848442 +0000 UTC m=+69.176014963" Nov 04 00:58:34.251713 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:34.251671 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-289kb" event={"ID":"9481376a-a63f-46e2-a605-fe4bb019c331","Type":"ContainerStarted","Data":"1b88ea9566ea7f4d202b4b9f0f68f710c1adfade65438f0544e74c89b17779b5"} Nov 04 00:58:34.252164 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:34.251784 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:58:34.270204 ip-10-0-128-129 kubenswrapper[2577]: I1104 00:58:34.270132 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-289kb" podStartSLOduration=67.626974506 podStartE2EDuration="1m11.270117145s" totalImagesPullingTime="3.643142639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:57:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:34.269550964 +0000 UTC m=+71.181717505" watchObservedRunningTime="2026-11-04 00:58:34.270117145 +0000 UTC m=+71.182283705" Nov 04 00:58:34.981450 ip-10-0-128-129 systemd[1]: Stopping Kubernetes Kubelet... Nov 04 00:58:34.995312 ip-10-0-128-129 systemd[1]: kubelet.service: Deactivated successfully. Nov 04 00:58:34.995501 ip-10-0-128-129 systemd[1]: Stopped Kubernetes Kubelet. Nov 04 00:58:34.995682 ip-10-0-128-129 systemd[1]: kubelet.service: Consumed 4.558s CPU time. Nov 04 00:58:35.009526 ip-10-0-128-129 systemd[1]: Starting Kubernetes Kubelet... Nov 04 00:58:35.087447 ip-10-0-128-129 kubenswrapper[7059]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 00:58:35.087447 ip-10-0-128-129 kubenswrapper[7059]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 04 00:58:35.087447 ip-10-0-128-129 kubenswrapper[7059]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 00:58:35.087447 ip-10-0-128-129 kubenswrapper[7059]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 00:58:35.089609 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089538 7059 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:58:35.089609 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089610 7059 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:58:35.089682 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089615 7059 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:58:35.089682 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089620 7059 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:58:35.089682 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089623 7059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:58:35.089682 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089626 7059 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:58:35.089682 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089629 7059 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:58:35.089682 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089631 7059 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:58:35.089682 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089635 7059 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:58:35.089682 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089637 7059 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:58:35.089682 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089640 7059 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:58:35.089682 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089643 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:58:35.089682 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089646 7059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:58:35.089682 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089648 7059 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:58:35.089682 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089651 7059 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:58:35.089682 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089653 7059 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:58:35.089682 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089656 7059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:58:35.089682 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089663 7059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:58:35.089682 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089667 7059 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:58:35.089682 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089669 7059 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:58:35.089682 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089672 7059 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:58:35.090173 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089675 7059 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:58:35.090173 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089677 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:58:35.090173 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089680 7059 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:58:35.090173 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089683 7059 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:58:35.090173 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089686 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:58:35.090173 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089689 7059 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:58:35.090173 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089692 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:58:35.090173 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089695 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:58:35.090173 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089698 7059 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:58:35.090173 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089700 7059 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:58:35.090173 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089703 7059 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:58:35.090173 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089705 7059 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:58:35.090173 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089708 7059 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:58:35.090173 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089710 7059 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:58:35.090173 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089713 7059 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:58:35.090173 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089716 7059 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:58:35.090173 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089719 7059 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:58:35.090173 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089722 7059 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:58:35.090173 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089724 7059 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:58:35.090173 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089727 7059 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:58:35.090681 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089729 7059 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:58:35.090681 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089732 7059 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:58:35.090681 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089735 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:58:35.090681 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089737 7059 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:58:35.090681 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089740 7059 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:58:35.090681 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089743 7059 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:58:35.090681 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089745 7059 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:58:35.090681 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089748 7059 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:58:35.090681 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089750 7059 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:58:35.090681 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089753 7059 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:58:35.090681 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089755 7059 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:58:35.090681 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089758 7059 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:58:35.090681 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089760 7059 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:58:35.090681 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089764 7059 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:58:35.090681 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089768 7059 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:58:35.090681 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089772 7059 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:58:35.090681 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089775 7059 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:58:35.090681 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089778 7059 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:58:35.090681 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089781 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:58:35.091197 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089783 7059 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:58:35.091197 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089786 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:58:35.091197 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089789 7059 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:58:35.091197 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089791 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:58:35.091197 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089794 7059 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:58:35.091197 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089797 7059 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:58:35.091197 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089800 7059 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:58:35.091197 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089802 7059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:58:35.091197 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089805 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:58:35.091197 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089807 7059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:58:35.091197 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089814 7059 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:58:35.091197 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089817 7059 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:58:35.091197 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089819 7059 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:58:35.091197 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089822 7059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:58:35.091197 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089824 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:58:35.091197 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089827 7059 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:58:35.091197 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089830 7059 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:58:35.091197 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089832 7059 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:58:35.091197 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089835 7059 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:58:35.091672 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089838 7059 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:58:35.091672 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089840 7059 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:58:35.091672 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089843 7059 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:58:35.091672 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089845 7059 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:58:35.091672 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089848 7059 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:58:35.091672 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089850 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:58:35.091672 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089853 7059 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:58:35.091672 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089856 7059 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:58:35.091672 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089859 7059 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:58:35.091672 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089861 7059 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:58:35.091672 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089864 7059 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:58:35.091672 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089867 7059 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:58:35.091672 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089869 7059 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:58:35.091672 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089872 7059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:58:35.091672 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089874 7059 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:58:35.091672 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089878 7059 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:58:35.091672 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089880 7059 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:58:35.091672 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089883 7059 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:58:35.091672 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089886 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:58:35.091672 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089888 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:58:35.092219 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089891 7059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:58:35.092219 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089895 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:58:35.092219 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089897 7059 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:58:35.092219 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089900 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:58:35.092219 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089903 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:58:35.092219 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089905 7059 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:58:35.092219 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.089908 7059 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:58:35.092219 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090635 7059 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:58:35.092219 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090642 7059 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:58:35.092219 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090645 7059 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:58:35.092219 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090648 7059 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:58:35.092219 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090651 7059 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:58:35.092219 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090653 7059 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:58:35.092219 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090656 7059 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:58:35.092219 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090659 7059 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:58:35.092219 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090661 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:58:35.092219 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090664 7059 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:58:35.092219 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090667 7059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:58:35.092219 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090669 7059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:58:35.092685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090673 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:58:35.092685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090676 7059 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:58:35.092685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090679 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:58:35.092685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090682 7059 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:58:35.092685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090687 7059 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:58:35.092685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090690 7059 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:58:35.092685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090693 7059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:58:35.092685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090696 7059 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:58:35.092685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090699 7059 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:58:35.092685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090702 7059 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:58:35.092685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090704 7059 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:58:35.092685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090707 7059 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:58:35.092685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090709 7059 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:58:35.092685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090712 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:58:35.092685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090714 7059 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:58:35.092685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090717 7059 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:58:35.092685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090720 7059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:58:35.092685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090722 7059 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:58:35.092685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090726 7059 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:58:35.093174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090728 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:58:35.093174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090731 7059 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:58:35.093174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090733 7059 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:58:35.093174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090736 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:58:35.093174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090739 7059 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:58:35.093174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090741 7059 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:58:35.093174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090744 7059 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:58:35.093174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090746 7059 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:58:35.093174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090749 7059 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:58:35.093174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090751 7059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:58:35.093174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090754 7059 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:58:35.093174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090756 7059 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:58:35.093174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090759 7059 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:58:35.093174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090762 7059 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:58:35.093174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090765 7059 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:58:35.093174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090767 7059 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:58:35.093174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090770 7059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:58:35.093174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090773 7059 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:58:35.093174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090775 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:58:35.093174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090778 7059 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:58:35.093670 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090781 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:58:35.093670 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090783 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:58:35.093670 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090786 7059 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:58:35.093670 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090788 7059 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:58:35.093670 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090791 7059 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:58:35.093670 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090793 7059 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:58:35.093670 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090796 7059 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:58:35.093670 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090798 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:58:35.093670 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090801 7059 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:58:35.093670 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090804 7059 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:58:35.093670 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090806 7059 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:58:35.093670 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090809 7059 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:58:35.093670 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090812 7059 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:58:35.093670 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090815 7059 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:58:35.093670 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090818 7059 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:58:35.093670 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090820 7059 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:58:35.093670 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090823 7059 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:58:35.093670 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090826 7059 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:58:35.093670 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090828 7059 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:58:35.093670 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090831 7059 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:58:35.094174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090833 7059 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:58:35.094174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090836 7059 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:58:35.094174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090838 7059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:58:35.094174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090841 7059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:58:35.094174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090843 7059 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:58:35.094174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090847 7059 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:58:35.094174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090849 7059 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:58:35.094174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090852 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:58:35.094174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090855 7059 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:58:35.094174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090858 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:58:35.094174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090863 7059 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:58:35.094174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090866 7059 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:58:35.094174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090869 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:58:35.094174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090872 7059 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:58:35.094174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090875 7059 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:58:35.094174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090878 7059 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:58:35.094174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090880 7059 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:58:35.094174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090883 7059 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:58:35.094174 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090886 7059 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:58:35.094685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090888 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:58:35.094685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090891 7059 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:58:35.094685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090893 7059 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:58:35.094685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090896 7059 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:58:35.094685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090898 7059 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:58:35.094685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090900 7059 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:58:35.094685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090903 7059 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:58:35.094685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090906 7059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:58:35.094685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090909 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:58:35.094685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090912 7059 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:58:35.094685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090914 7059 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:58:35.094685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090917 7059 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:58:35.094685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090934 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:58:35.094685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090936 7059 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:58:35.094685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090939 7059 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:58:35.094685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.090942 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:58:35.094685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091447 7059 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:58:35.094685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091454 7059 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:58:35.094685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091458 7059 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:58:35.094685 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091461 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:58:35.095195 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091464 7059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:58:35.095195 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091467 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:58:35.095195 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091469 7059 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:58:35.095195 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091472 7059 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:58:35.095195 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091475 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:58:35.095195 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091477 7059 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:58:35.095195 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091479 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:58:35.095195 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091482 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:58:35.095195 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091485 7059 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:58:35.095195 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091487 7059 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:58:35.095195 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091490 7059 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:58:35.095195 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091492 7059 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:58:35.095195 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091495 7059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:58:35.095195 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091497 7059 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:58:35.095195 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091500 7059 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:58:35.095195 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091503 7059 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:58:35.095195 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091506 7059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:58:35.095195 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091508 7059 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:58:35.095195 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091511 7059 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:58:35.095866 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091515 7059 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:58:35.095866 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091519 7059 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:58:35.095866 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091522 7059 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:58:35.095866 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091525 7059 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:58:35.095866 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091528 7059 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:58:35.095866 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091531 7059 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:58:35.095866 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091534 7059 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:58:35.095866 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091536 7059 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:58:35.095866 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091539 7059 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:58:35.095866 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091542 7059 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:58:35.095866 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091544 7059 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:58:35.095866 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091548 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:58:35.095866 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091550 7059 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:58:35.095866 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091553 7059 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:58:35.095866 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091556 7059 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:58:35.095866 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091558 7059 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:58:35.095866 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091561 7059 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:58:35.095866 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091564 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:58:35.095866 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091566 7059 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:58:35.096689 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091569 7059 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:58:35.096689 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091571 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:58:35.096689 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091574 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:58:35.096689 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091576 7059 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:58:35.096689 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091579 7059 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:58:35.096689 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091582 7059 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:58:35.096689 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091584 7059 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:58:35.096689 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091587 7059 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:58:35.096689 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091590 7059 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:58:35.096689 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091592 7059 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:58:35.096689 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091595 7059 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:58:35.096689 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091597 7059 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:58:35.096689 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091600 7059 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:58:35.096689 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091603 7059 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:58:35.096689 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091606 7059 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:58:35.096689 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091608 7059 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:58:35.096689 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091611 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:58:35.096689 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091614 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:58:35.096689 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091616 7059 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:58:35.096689 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091619 7059 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:58:35.097249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091621 7059 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:58:35.097249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091624 7059 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:58:35.097249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091626 7059 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:58:35.097249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091629 7059 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:58:35.097249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091631 7059 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:58:35.097249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091634 7059 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:58:35.097249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091637 7059 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:58:35.097249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091639 7059 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:58:35.097249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091642 7059 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:58:35.097249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091646 7059 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:58:35.097249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091649 7059 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:58:35.097249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091651 7059 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:58:35.097249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091654 7059 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:58:35.097249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091656 7059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:58:35.097249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091659 7059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:58:35.097249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091661 7059 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:58:35.097249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091665 7059 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:58:35.097249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091668 7059 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:58:35.097249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091671 7059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:58:35.097713 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091674 7059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:58:35.097713 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091678 7059 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:58:35.097713 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091680 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:58:35.097713 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091683 7059 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:58:35.097713 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091686 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:58:35.097713 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091690 7059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:58:35.097713 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091692 7059 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:58:35.097713 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091696 7059 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:58:35.097713 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091699 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:58:35.097713 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091701 7059 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:58:35.097713 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091704 7059 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:58:35.097713 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091706 7059 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:58:35.097713 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091709 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:58:35.097713 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091712 7059 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:58:35.097713 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091714 7059 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:58:35.097713 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091717 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:58:35.097713 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091719 7059 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:58:35.097713 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091722 7059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:58:35.097713 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091725 7059 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:58:35.097713 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091727 7059 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:58:35.098222 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091731 7059 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:58:35.098222 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091733 7059 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:58:35.098222 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091738 7059 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:58:35.098222 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091740 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:58:35.098222 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.091743 7059 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:58:35.098222 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091830 7059 flags.go:64] FLAG: --address="0.0.0.0" Nov 04 00:58:35.098222 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091841 7059 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 04 00:58:35.098222 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091847 7059 flags.go:64] FLAG: --anonymous-auth="true" Nov 04 00:58:35.098222 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091852 7059 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 04 00:58:35.098222 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091856 7059 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 04 00:58:35.098222 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091859 7059 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 04 00:58:35.098222 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091863 7059 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 04 00:58:35.098222 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091867 7059 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 04 00:58:35.098222 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091870 7059 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 04 00:58:35.098222 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091873 7059 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 04 00:58:35.098222 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091876 7059 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 04 00:58:35.098222 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091880 7059 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 04 00:58:35.098222 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091886 7059 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 04 00:58:35.098222 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091889 7059 flags.go:64] FLAG: --cgroup-root="" Nov 04 00:58:35.098222 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091892 7059 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 04 00:58:35.098222 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091895 7059 flags.go:64] FLAG: --client-ca-file="" Nov 04 00:58:35.098222 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091898 7059 flags.go:64] FLAG: --cloud-provider="external" Nov 04 00:58:35.098222 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091901 7059 flags.go:64] FLAG: --cluster-dns="[]" Nov 04 00:58:35.098789 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091905 7059 flags.go:64] FLAG: --cluster-domain="" Nov 04 00:58:35.098789 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091908 7059 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 04 00:58:35.098789 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091911 7059 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 04 00:58:35.098789 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091914 7059 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 04 00:58:35.098789 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091917 7059 flags.go:64] FLAG: --container-log-max-files="5" Nov 04 00:58:35.098789 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091936 7059 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 04 00:58:35.098789 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091939 7059 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 04 00:58:35.098789 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091942 7059 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 04 00:58:35.098789 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091945 7059 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 04 00:58:35.098789 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091948 7059 flags.go:64] FLAG: --contention-profiling="false" Nov 04 00:58:35.098789 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091951 7059 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 04 00:58:35.098789 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091955 7059 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 04 00:58:35.098789 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091958 7059 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 04 00:58:35.098789 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091961 7059 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 04 00:58:35.098789 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091966 7059 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 04 00:58:35.098789 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091969 7059 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 04 00:58:35.098789 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091972 7059 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 04 00:58:35.098789 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091975 7059 flags.go:64] FLAG: --enable-load-reader="false" Nov 04 00:58:35.098789 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091978 7059 flags.go:64] FLAG: --enable-server="true" Nov 04 00:58:35.098789 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091981 7059 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 04 00:58:35.098789 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091991 7059 flags.go:64] FLAG: --event-burst="100" Nov 04 00:58:35.098789 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091994 7059 flags.go:64] FLAG: --event-qps="50" Nov 04 00:58:35.098789 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.091997 7059 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 04 00:58:35.098789 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092000 7059 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 04 00:58:35.099399 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092003 7059 flags.go:64] FLAG: --eviction-hard="" Nov 04 00:58:35.099399 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092011 7059 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 04 00:58:35.099399 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092016 7059 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 04 00:58:35.099399 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092019 7059 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 04 00:58:35.099399 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092022 7059 flags.go:64] FLAG: --eviction-soft="" Nov 04 00:58:35.099399 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092025 7059 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 04 00:58:35.099399 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092029 7059 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 04 00:58:35.099399 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092031 7059 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 04 00:58:35.099399 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092038 7059 flags.go:64] FLAG: --experimental-mounter-path="" Nov 04 00:58:35.099399 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092041 7059 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 04 00:58:35.099399 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092044 7059 flags.go:64] FLAG: --fail-swap-on="true" Nov 04 00:58:35.099399 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092047 7059 flags.go:64] FLAG: --feature-gates="" Nov 04 00:58:35.099399 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092051 7059 flags.go:64] FLAG: --file-check-frequency="20s" Nov 04 00:58:35.099399 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092054 7059 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 04 00:58:35.099399 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092057 7059 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 04 00:58:35.099399 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092060 7059 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 04 00:58:35.099399 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092063 7059 flags.go:64] FLAG: --healthz-port="10248" Nov 04 00:58:35.099399 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092066 7059 flags.go:64] FLAG: --help="false" Nov 04 00:58:35.099399 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092069 7059 flags.go:64] FLAG: --hostname-override="ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:58:35.099399 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092074 7059 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 04 00:58:35.099399 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092077 7059 flags.go:64] FLAG: --http-check-frequency="20s" Nov 04 00:58:35.099399 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092080 7059 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 04 00:58:35.099399 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092083 7059 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 04 00:58:35.099998 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092087 7059 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 04 00:58:35.099998 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092090 7059 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 04 00:58:35.099998 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092092 7059 flags.go:64] FLAG: --image-service-endpoint="" Nov 04 00:58:35.099998 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092095 7059 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 04 00:58:35.099998 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092098 7059 flags.go:64] FLAG: --kube-api-burst="100" Nov 04 00:58:35.099998 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092101 7059 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 04 00:58:35.099998 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092104 7059 flags.go:64] FLAG: --kube-api-qps="50" Nov 04 00:58:35.099998 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092107 7059 flags.go:64] FLAG: --kube-reserved="" Nov 04 00:58:35.099998 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092110 7059 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 04 00:58:35.099998 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092113 7059 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 04 00:58:35.099998 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092116 7059 flags.go:64] FLAG: --kubelet-cgroups="" Nov 04 00:58:35.099998 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092120 7059 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 04 00:58:35.099998 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092123 7059 flags.go:64] FLAG: --lock-file="" Nov 04 00:58:35.099998 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092125 7059 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 04 00:58:35.099998 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092128 7059 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 04 00:58:35.099998 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092131 7059 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 04 00:58:35.099998 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092137 7059 flags.go:64] FLAG: --log-json-split-stream="false" Nov 04 00:58:35.099998 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092140 7059 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 04 00:58:35.099998 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092143 7059 flags.go:64] FLAG: --log-text-split-stream="false" Nov 04 00:58:35.099998 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092146 7059 flags.go:64] FLAG: --logging-format="text" Nov 04 00:58:35.099998 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092149 7059 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 04 00:58:35.099998 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092152 7059 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 04 00:58:35.099998 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092155 7059 flags.go:64] FLAG: --manifest-url="" Nov 04 00:58:35.099998 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092158 7059 flags.go:64] FLAG: --manifest-url-header="" Nov 04 00:58:35.099998 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092162 7059 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 04 00:58:35.100643 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092165 7059 flags.go:64] FLAG: --max-open-files="1000000" Nov 04 00:58:35.100643 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092169 7059 flags.go:64] FLAG: --max-pods="110" Nov 04 00:58:35.100643 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092172 7059 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 04 00:58:35.100643 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092177 7059 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 04 00:58:35.100643 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092180 7059 flags.go:64] FLAG: --memory-manager-policy="None" Nov 04 00:58:35.100643 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092183 7059 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 04 00:58:35.100643 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092186 7059 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 04 00:58:35.100643 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092189 7059 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 04 00:58:35.100643 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092192 7059 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 04 00:58:35.100643 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092198 7059 flags.go:64] FLAG: --node-status-max-images="50" Nov 04 00:58:35.100643 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092201 7059 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 04 00:58:35.100643 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092204 7059 flags.go:64] FLAG: --oom-score-adj="-999" Nov 04 00:58:35.100643 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092207 7059 flags.go:64] FLAG: --pod-cidr="" Nov 04 00:58:35.100643 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092210 7059 flags.go:64] FLAG: --pod-manifest-path="" Nov 04 00:58:35.100643 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092213 7059 flags.go:64] FLAG: --pod-max-pids="-1" Nov 04 00:58:35.100643 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092216 7059 flags.go:64] FLAG: --pods-per-core="0" Nov 04 00:58:35.100643 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092219 7059 flags.go:64] FLAG: --port="10250" Nov 04 00:58:35.100643 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092222 7059 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 04 00:58:35.100643 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092226 7059 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-000c8fb1108dc6043" Nov 04 00:58:35.100643 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092230 7059 flags.go:64] FLAG: --qos-reserved="" Nov 04 00:58:35.100643 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092233 7059 flags.go:64] FLAG: --read-only-port="10255" Nov 04 00:58:35.100643 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092236 7059 flags.go:64] FLAG: --register-node="true" Nov 04 00:58:35.100643 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092239 7059 flags.go:64] FLAG: --register-with-taints="" Nov 04 00:58:35.100643 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092243 7059 flags.go:64] FLAG: --registry-burst="10" Nov 04 00:58:35.100643 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092246 7059 flags.go:64] FLAG: --registry-qps="5" Nov 04 00:58:35.100643 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092249 7059 flags.go:64] FLAG: --reserved-cpus="" Nov 04 00:58:35.101291 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092252 7059 flags.go:64] FLAG: --reserved-memory="" Nov 04 00:58:35.101291 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092255 7059 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 04 00:58:35.101291 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092258 7059 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 04 00:58:35.101291 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092262 7059 flags.go:64] FLAG: --rotate-certificates="false" Nov 04 00:58:35.101291 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092264 7059 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 04 00:58:35.101291 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092267 7059 flags.go:64] FLAG: --runonce="false" Nov 04 00:58:35.101291 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092270 7059 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 04 00:58:35.101291 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092273 7059 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 04 00:58:35.101291 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092276 7059 flags.go:64] FLAG: --seccomp-default="false" Nov 04 00:58:35.101291 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092280 7059 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 04 00:58:35.101291 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092283 7059 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 04 00:58:35.101291 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092286 7059 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 04 00:58:35.101291 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092289 7059 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 04 00:58:35.101291 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092292 7059 flags.go:64] FLAG: --storage-driver-password="root" Nov 04 00:58:35.101291 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092294 7059 flags.go:64] FLAG: --storage-driver-secure="false" Nov 04 00:58:35.101291 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092297 7059 flags.go:64] FLAG: --storage-driver-table="stats" Nov 04 00:58:35.101291 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092300 7059 flags.go:64] FLAG: --storage-driver-user="root" Nov 04 00:58:35.101291 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092303 7059 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 04 00:58:35.101291 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092306 7059 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 04 00:58:35.101291 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092309 7059 flags.go:64] FLAG: --system-cgroups="" Nov 04 00:58:35.101291 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092311 7059 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 04 00:58:35.101291 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092316 7059 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 04 00:58:35.101291 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092319 7059 flags.go:64] FLAG: --tls-cert-file="" Nov 04 00:58:35.101291 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092322 7059 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 04 00:58:35.101291 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092327 7059 flags.go:64] FLAG: --tls-min-version="" Nov 04 00:58:35.101899 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092330 7059 flags.go:64] FLAG: --tls-private-key-file="" Nov 04 00:58:35.101899 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092332 7059 flags.go:64] FLAG: --topology-manager-policy="none" Nov 04 00:58:35.101899 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092335 7059 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 04 00:58:35.101899 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092338 7059 flags.go:64] FLAG: --topology-manager-scope="container" Nov 04 00:58:35.101899 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092341 7059 flags.go:64] FLAG: --v="2" Nov 04 00:58:35.101899 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092345 7059 flags.go:64] FLAG: --version="false" Nov 04 00:58:35.101899 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092349 7059 flags.go:64] FLAG: --vmodule="" Nov 04 00:58:35.101899 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092353 7059 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 04 00:58:35.101899 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092357 7059 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 04 00:58:35.101899 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092481 7059 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:58:35.101899 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092484 7059 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:58:35.101899 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092487 7059 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:58:35.101899 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092491 7059 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:58:35.101899 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092494 7059 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:58:35.101899 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092496 7059 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:58:35.101899 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092499 7059 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:58:35.101899 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092503 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:58:35.101899 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092506 7059 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:58:35.101899 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092508 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:58:35.101899 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092511 7059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:58:35.101899 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092513 7059 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:58:35.102435 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092517 7059 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:58:35.102435 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092521 7059 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:58:35.102435 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092524 7059 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:58:35.102435 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092527 7059 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:58:35.102435 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092531 7059 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:58:35.102435 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092534 7059 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:58:35.102435 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092537 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:58:35.102435 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092540 7059 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:58:35.102435 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092543 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:58:35.102435 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092545 7059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:58:35.102435 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092549 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:58:35.102435 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092552 7059 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:58:35.102435 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092555 7059 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:58:35.102435 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092557 7059 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:58:35.102435 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092560 7059 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:58:35.102435 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092563 7059 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:58:35.102435 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092565 7059 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:58:35.102435 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092568 7059 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:58:35.102435 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092571 7059 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:58:35.102989 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092574 7059 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:58:35.102989 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092576 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:58:35.102989 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092579 7059 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:58:35.102989 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092582 7059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:58:35.102989 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092585 7059 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:58:35.102989 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092587 7059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:58:35.102989 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092590 7059 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:58:35.102989 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092593 7059 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:58:35.102989 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092597 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:58:35.102989 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092599 7059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:58:35.102989 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092602 7059 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:58:35.102989 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092604 7059 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:58:35.102989 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092607 7059 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:58:35.102989 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092609 7059 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:58:35.102989 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092612 7059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:58:35.102989 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092614 7059 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:58:35.102989 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092617 7059 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:58:35.102989 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092619 7059 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:58:35.102989 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092622 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:58:35.103589 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092624 7059 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:58:35.103589 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092627 7059 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:58:35.103589 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092630 7059 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:58:35.103589 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092633 7059 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:58:35.103589 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092636 7059 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:58:35.103589 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092639 7059 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:58:35.103589 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092642 7059 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:58:35.103589 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092645 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:58:35.103589 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092647 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:58:35.103589 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092650 7059 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:58:35.103589 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092652 7059 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:58:35.103589 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092655 7059 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:58:35.103589 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092657 7059 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:58:35.103589 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092661 7059 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:58:35.103589 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092663 7059 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:58:35.103589 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092666 7059 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:58:35.103589 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092669 7059 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:58:35.103589 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092671 7059 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:58:35.103589 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092674 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:58:35.103589 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092676 7059 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:58:35.104312 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092679 7059 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:58:35.104312 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092682 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:58:35.104312 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092685 7059 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:58:35.104312 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092688 7059 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:58:35.104312 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092690 7059 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:58:35.104312 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092693 7059 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:58:35.104312 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092696 7059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:58:35.104312 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092699 7059 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:58:35.104312 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092701 7059 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:58:35.104312 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092704 7059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:58:35.104312 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092706 7059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:58:35.104312 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092709 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:58:35.104312 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092711 7059 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:58:35.104312 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092714 7059 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:58:35.104312 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092716 7059 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:58:35.104312 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092719 7059 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:58:35.104312 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092723 7059 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:58:35.104312 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092725 7059 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:58:35.104312 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092728 7059 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:58:35.105168 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092731 7059 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:58:35.105168 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092733 7059 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:58:35.105168 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092736 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:58:35.105168 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092738 7059 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:58:35.105168 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092741 7059 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:58:35.105168 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092744 7059 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:58:35.105168 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092746 7059 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:58:35.105168 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092749 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:58:35.105168 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092752 7059 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:58:35.105168 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092754 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:58:35.105168 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092757 7059 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:58:35.105168 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092759 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:58:35.105168 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092762 7059 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:58:35.105168 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092765 7059 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:58:35.105168 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092768 7059 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:58:35.105168 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092771 7059 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:58:35.105168 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092773 7059 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:58:35.105945 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.092798 7059 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 00:58:35.105945 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092833 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:58:35.105945 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092837 7059 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:58:35.105945 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092840 7059 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:58:35.105945 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092843 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:58:35.105945 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092845 7059 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:58:35.105945 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092848 7059 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:58:35.105945 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092851 7059 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:58:35.105945 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092853 7059 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:58:35.105945 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092856 7059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:58:35.105945 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092859 7059 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:58:35.105945 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092861 7059 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:58:35.105945 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092864 7059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:58:35.105945 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092867 7059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:58:35.105945 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092870 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:58:35.106496 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092872 7059 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:58:35.106496 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092875 7059 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:58:35.106496 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092878 7059 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:58:35.106496 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092880 7059 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:58:35.106496 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092883 7059 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:58:35.106496 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092885 7059 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:58:35.106496 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092888 7059 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:58:35.106496 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092890 7059 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:58:35.106496 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092895 7059 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:58:35.106496 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092897 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:58:35.106496 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092900 7059 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:58:35.106496 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092902 7059 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:58:35.106496 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092905 7059 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:58:35.106496 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092908 7059 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:58:35.106496 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092911 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:58:35.106496 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092914 7059 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:58:35.106496 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092916 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:58:35.106496 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092932 7059 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:58:35.106496 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092936 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:58:35.106496 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092938 7059 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:58:35.107200 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092941 7059 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:58:35.107200 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092943 7059 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:58:35.107200 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092946 7059 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:58:35.107200 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092948 7059 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:58:35.107200 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092951 7059 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:58:35.107200 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092953 7059 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:58:35.107200 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092956 7059 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:58:35.107200 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092958 7059 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:58:35.107200 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092961 7059 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:58:35.107200 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092963 7059 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:58:35.107200 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092967 7059 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:58:35.107200 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092969 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:58:35.107200 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092972 7059 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:58:35.107200 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092975 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:58:35.107200 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092977 7059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:58:35.107200 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092980 7059 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:58:35.107200 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092984 7059 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:58:35.107200 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092988 7059 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:58:35.107200 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.092991 7059 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:58:35.107729 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093008 7059 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:58:35.107729 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093012 7059 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:58:35.107729 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093017 7059 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:58:35.107729 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093021 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:58:35.107729 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093025 7059 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:58:35.107729 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093028 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:58:35.107729 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093031 7059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:58:35.107729 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093035 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:58:35.107729 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093038 7059 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:58:35.107729 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093041 7059 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:58:35.107729 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093043 7059 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:58:35.107729 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093046 7059 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:58:35.107729 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093048 7059 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:58:35.107729 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093051 7059 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:58:35.107729 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093054 7059 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:58:35.107729 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093056 7059 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:58:35.107729 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093059 7059 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:58:35.107729 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093061 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:58:35.107729 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093064 7059 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:58:35.107729 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093066 7059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:58:35.108243 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093069 7059 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:58:35.108243 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093072 7059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:58:35.108243 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093074 7059 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:58:35.108243 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093078 7059 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:58:35.108243 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093080 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:58:35.108243 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093083 7059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:58:35.108243 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093086 7059 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:58:35.108243 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093088 7059 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:58:35.108243 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093091 7059 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:58:35.108243 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093093 7059 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:58:35.108243 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093096 7059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:58:35.108243 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093099 7059 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:58:35.108243 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093101 7059 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:58:35.108243 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093104 7059 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:58:35.108243 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093106 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:58:35.108243 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093109 7059 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:58:35.108243 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093112 7059 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:58:35.108243 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093115 7059 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:58:35.108243 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093117 7059 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:58:35.108243 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093121 7059 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:58:35.108747 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093124 7059 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:58:35.108747 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093127 7059 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:58:35.108747 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093129 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:58:35.108747 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093131 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:58:35.108747 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093134 7059 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:58:35.108747 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093136 7059 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:58:35.108747 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093139 7059 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:58:35.108747 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093141 7059 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:58:35.108747 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093144 7059 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:58:35.108747 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093146 7059 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:58:35.108747 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093149 7059 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:58:35.108747 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093153 7059 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:58:35.108747 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.093155 7059 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:58:35.108747 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.098310 7059 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 04 00:58:35.108747 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.098327 7059 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 04 00:58:35.108747 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.098346 7059 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 04 00:58:35.108747 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.098350 7059 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 04 00:58:35.108747 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098413 7059 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:58:35.108747 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098418 7059 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:58:35.108747 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098421 7059 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:58:35.109249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098424 7059 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:58:35.109249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098427 7059 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:58:35.109249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098430 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:58:35.109249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098432 7059 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:58:35.109249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098435 7059 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:58:35.109249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098438 7059 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:58:35.109249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098441 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:58:35.109249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098444 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:58:35.109249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098446 7059 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:58:35.109249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098449 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:58:35.109249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098452 7059 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:58:35.109249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098455 7059 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:58:35.109249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098457 7059 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:58:35.109249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098461 7059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:58:35.109249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098463 7059 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:58:35.109249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098466 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:58:35.109249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098469 7059 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:58:35.109249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098471 7059 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:58:35.109249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098474 7059 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:58:35.109249 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098477 7059 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:58:35.109751 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098479 7059 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:58:35.109751 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098482 7059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:58:35.109751 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098484 7059 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:58:35.109751 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098487 7059 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:58:35.109751 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098490 7059 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:58:35.109751 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098493 7059 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:58:35.109751 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098496 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:58:35.109751 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098499 7059 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:58:35.109751 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098502 7059 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:58:35.109751 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098505 7059 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:58:35.109751 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098507 7059 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:58:35.109751 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098510 7059 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:58:35.109751 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098512 7059 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:58:35.109751 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098515 7059 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:58:35.109751 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098517 7059 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:58:35.109751 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098520 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:58:35.109751 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098522 7059 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:58:35.109751 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098525 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:58:35.109751 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098528 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:58:35.110397 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098530 7059 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:58:35.110397 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098533 7059 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:58:35.110397 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098537 7059 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:58:35.110397 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098541 7059 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:58:35.110397 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098544 7059 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:58:35.110397 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098547 7059 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:58:35.110397 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098551 7059 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:58:35.110397 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098555 7059 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:58:35.110397 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098558 7059 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:58:35.110397 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098561 7059 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:58:35.110397 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098564 7059 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:58:35.110397 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098568 7059 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:58:35.110397 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098571 7059 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:58:35.110397 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098574 7059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:58:35.110397 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098576 7059 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:58:35.110397 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098579 7059 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:58:35.110397 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098582 7059 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:58:35.110397 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098585 7059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:58:35.110397 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098589 7059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:58:35.110870 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098591 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:58:35.110870 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098602 7059 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:58:35.110870 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098605 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:58:35.110870 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098607 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:58:35.110870 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098610 7059 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:58:35.110870 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098613 7059 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:58:35.110870 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098616 7059 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:58:35.110870 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098618 7059 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:58:35.110870 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098621 7059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:58:35.110870 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098624 7059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:58:35.110870 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098626 7059 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:58:35.110870 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098629 7059 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:58:35.110870 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098631 7059 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:58:35.110870 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098634 7059 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:58:35.110870 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098637 7059 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:58:35.110870 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098639 7059 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:58:35.110870 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098642 7059 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:58:35.110870 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098644 7059 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:58:35.110870 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098648 7059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:58:35.110870 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098650 7059 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:58:35.111500 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098653 7059 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:58:35.111500 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098655 7059 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:58:35.111500 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098658 7059 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:58:35.111500 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098660 7059 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:58:35.111500 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098663 7059 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:58:35.111500 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098666 7059 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:58:35.111500 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098669 7059 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:58:35.111500 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098671 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:58:35.111500 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098674 7059 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:58:35.111500 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098678 7059 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:58:35.111500 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098680 7059 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:58:35.111500 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098683 7059 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:58:35.111500 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098686 7059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:58:35.111500 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098688 7059 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:58:35.111500 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098691 7059 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:58:35.111500 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098693 7059 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:58:35.111500 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098696 7059 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:58:35.111500 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098699 7059 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:58:35.111500 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098701 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:58:35.112075 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098704 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:58:35.112075 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098706 7059 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:58:35.112075 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098709 7059 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:58:35.112075 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098711 7059 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:58:35.112075 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098714 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:58:35.112075 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098717 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:58:35.112075 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.098744 7059 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 00:58:35.112075 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098890 7059 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:58:35.112075 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098895 7059 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:58:35.112075 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098898 7059 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:58:35.112075 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098901 7059 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:58:35.112075 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098903 7059 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:58:35.112075 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098907 7059 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:58:35.112075 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098909 7059 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:58:35.112075 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098912 7059 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:58:35.112075 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098914 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:58:35.112473 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098917 7059 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:58:35.112473 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098936 7059 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:58:35.112473 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098940 7059 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:58:35.112473 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098942 7059 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:58:35.112473 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098945 7059 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:58:35.112473 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098948 7059 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:58:35.112473 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098951 7059 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:58:35.112473 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098954 7059 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:58:35.112473 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098956 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:58:35.112473 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098959 7059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:58:35.112473 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098962 7059 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:58:35.112473 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098964 7059 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:58:35.112473 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098967 7059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:58:35.112473 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098970 7059 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:58:35.112473 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098973 7059 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:58:35.112473 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098975 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:58:35.112473 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098978 7059 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:58:35.112473 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098981 7059 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:58:35.112473 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098983 7059 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:58:35.112473 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098986 7059 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:58:35.113198 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098988 7059 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:58:35.113198 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098991 7059 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:58:35.113198 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098993 7059 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:58:35.113198 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098995 7059 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:58:35.113198 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.098998 7059 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:58:35.113198 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099001 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:58:35.113198 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099003 7059 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:58:35.113198 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099006 7059 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:58:35.113198 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099008 7059 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:58:35.113198 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099013 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:58:35.113198 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099016 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:58:35.113198 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099018 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:58:35.113198 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099021 7059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:58:35.113198 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099023 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:58:35.113198 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099026 7059 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:58:35.113198 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099028 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:58:35.113198 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099031 7059 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:58:35.113198 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099033 7059 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:58:35.113198 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099036 7059 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:58:35.114002 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099039 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:58:35.114002 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099041 7059 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:58:35.114002 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099044 7059 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:58:35.114002 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099046 7059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:58:35.114002 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099048 7059 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:58:35.114002 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099051 7059 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:58:35.114002 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099054 7059 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:58:35.114002 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099057 7059 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:58:35.114002 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099061 7059 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:58:35.114002 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099064 7059 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:58:35.114002 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099066 7059 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:58:35.114002 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099069 7059 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:58:35.114002 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099072 7059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:58:35.114002 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099074 7059 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:58:35.114002 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099077 7059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:58:35.114002 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099080 7059 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:58:35.114002 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099082 7059 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:58:35.114002 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099085 7059 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:58:35.114002 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099088 7059 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:58:35.114725 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099090 7059 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:58:35.114725 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099093 7059 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:58:35.114725 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099095 7059 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:58:35.114725 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099098 7059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:58:35.114725 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099101 7059 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:58:35.114725 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099104 7059 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:58:35.114725 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099106 7059 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:58:35.114725 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099109 7059 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:58:35.114725 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099112 7059 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:58:35.114725 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099114 7059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:58:35.114725 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099117 7059 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:58:35.114725 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099119 7059 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:58:35.114725 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099122 7059 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:58:35.114725 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099125 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:58:35.114725 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099128 7059 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:58:35.114725 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099132 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:58:35.114725 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099135 7059 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:58:35.114725 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099138 7059 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:58:35.114725 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099141 7059 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:58:35.114725 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099144 7059 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:58:35.115609 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099146 7059 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:58:35.115609 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099149 7059 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:58:35.115609 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099151 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:58:35.115609 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099154 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:58:35.115609 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099156 7059 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:58:35.115609 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099159 7059 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:58:35.115609 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099161 7059 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:58:35.115609 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099164 7059 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:58:35.115609 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099167 7059 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:58:35.115609 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099169 7059 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:58:35.115609 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099172 7059 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:58:35.115609 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099175 7059 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:58:35.115609 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099177 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:58:35.115609 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099179 7059 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:58:35.115609 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099182 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:58:35.115609 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099185 7059 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:58:35.115609 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099187 7059 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:58:35.115609 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099190 7059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:58:35.115609 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099193 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:58:35.116577 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.099217 7059 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 00:58:35.116577 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099251 7059 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:58:35.116577 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099256 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:58:35.116577 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099260 7059 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:58:35.116577 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099263 7059 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:58:35.116577 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099266 7059 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:58:35.116577 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099269 7059 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:58:35.116577 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099271 7059 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:58:35.116577 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099274 7059 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:58:35.116577 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099276 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:58:35.116577 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099279 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:58:35.116577 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099282 7059 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:58:35.116577 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099284 7059 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:58:35.116577 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099287 7059 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:58:35.116577 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099289 7059 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:58:35.117331 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099292 7059 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:58:35.117331 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099295 7059 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:58:35.117331 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099297 7059 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:58:35.117331 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099300 7059 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:58:35.117331 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099302 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:58:35.117331 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099305 7059 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:58:35.117331 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099307 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:58:35.117331 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099310 7059 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:58:35.117331 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099313 7059 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:58:35.117331 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099315 7059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:58:35.117331 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099318 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:58:35.117331 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099320 7059 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:58:35.117331 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099323 7059 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:58:35.117331 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099325 7059 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:58:35.117331 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099328 7059 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:58:35.117331 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099330 7059 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:58:35.117331 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099333 7059 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:58:35.117331 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099336 7059 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:58:35.117331 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099338 7059 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:58:35.118359 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099341 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:58:35.118359 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099344 7059 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:58:35.118359 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099346 7059 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:58:35.118359 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099349 7059 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:58:35.118359 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099352 7059 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:58:35.118359 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099354 7059 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:58:35.118359 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099357 7059 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:58:35.118359 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099359 7059 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:58:35.118359 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099362 7059 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:58:35.118359 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099365 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:58:35.118359 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099367 7059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:58:35.118359 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099369 7059 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:58:35.118359 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099372 7059 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:58:35.118359 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099374 7059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:58:35.118359 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099377 7059 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:58:35.118359 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099379 7059 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:58:35.118359 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099382 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:58:35.118359 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099384 7059 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:58:35.118359 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099387 7059 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:58:35.118359 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099389 7059 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:58:35.119412 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099392 7059 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:58:35.119412 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099394 7059 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:58:35.119412 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099396 7059 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:58:35.119412 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099399 7059 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:58:35.119412 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099402 7059 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:58:35.119412 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099404 7059 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:58:35.119412 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099407 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:58:35.119412 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099412 7059 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:58:35.119412 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099414 7059 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:58:35.119412 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099417 7059 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:58:35.119412 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099421 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:58:35.119412 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099424 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:58:35.119412 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099426 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:58:35.119412 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099429 7059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:58:35.119412 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099432 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:58:35.119412 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099435 7059 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:58:35.119412 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099437 7059 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:58:35.119412 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099440 7059 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:58:35.119412 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099442 7059 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:58:35.120324 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099445 7059 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:58:35.120324 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099447 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:58:35.120324 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099450 7059 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:58:35.120324 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099452 7059 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:58:35.120324 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099455 7059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:58:35.120324 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099457 7059 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:58:35.120324 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099460 7059 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:58:35.120324 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099462 7059 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:58:35.120324 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099467 7059 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:58:35.120324 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099470 7059 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:58:35.120324 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099473 7059 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:58:35.120324 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099475 7059 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:58:35.120324 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099478 7059 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:58:35.120324 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099480 7059 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:58:35.120324 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099483 7059 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:58:35.120324 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099485 7059 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:58:35.120324 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099488 7059 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:58:35.120324 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099491 7059 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:58:35.121224 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099493 7059 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:58:35.121224 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099496 7059 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:58:35.121224 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099499 7059 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:58:35.121224 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099502 7059 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:58:35.121224 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099504 7059 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:58:35.121224 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099507 7059 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:58:35.121224 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099510 7059 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:58:35.121224 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099513 7059 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:58:35.121224 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099516 7059 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:58:35.121224 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099518 7059 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:58:35.121224 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099522 7059 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:58:35.121224 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099524 7059 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:58:35.121224 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099527 7059 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:58:35.121224 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099529 7059 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:58:35.121224 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099532 7059 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:58:35.121224 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:35.099534 7059 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:58:35.121224 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.099593 7059 server.go:957] "Client rotation is on, will bootstrap in background" Nov 04 00:58:35.121224 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.100517 7059 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 04 00:58:35.121224 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.100572 7059 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 04 00:58:35.121224 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.100741 7059 server.go:1014] "Starting client certificate rotation" Nov 04 00:58:35.122095 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.100883 7059 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 00:58:35.122095 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.100948 7059 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-03 00:52:23 +0000 UTC" deadline="2028-05-26 14:19:43.62956988 +0000 UTC" Nov 04 00:58:35.122095 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.100989 7059 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13669h21m8.528582249s" Nov 04 00:58:35.122095 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.101289 7059 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 00:58:35.122095 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.102362 7059 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 00:58:35.122095 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.103805 7059 log.go:25] "Validated CRI v1 runtime API" Nov 04 00:58:35.122095 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.106303 7059 log.go:25] "Validated CRI v1 image API" Nov 04 00:58:35.122095 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.107016 7059 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 04 00:58:35.122095 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.109421 7059 fs.go:135] Filesystem UUIDs: map[780366c4-b376-4f0c-997d-64ac6d5ea20e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 7df6de30-b3da-4ecd-bcdf-8c2f3952a962:/dev/nvme0n1p3] Nov 04 00:58:35.122537 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.109440 7059 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0ee3f067670f8fcf04bafdaf8967f414cf98590768a1670fa6221c71f63157d6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0ee3f067670f8fcf04bafdaf8967f414cf98590768a1670fa6221c71f63157d6/userdata/shm major:0 minor:387 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/288b26a3d92fef361cf864edee46bc80d706a3065e7800ba294dcfc9d2c80a80/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/288b26a3d92fef361cf864edee46bc80d706a3065e7800ba294dcfc9d2c80a80/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d2525aeb17bd4389865536271eac885aad37b39e20e47e47da5fbc206bb7099/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d2525aeb17bd4389865536271eac885aad37b39e20e47e47da5fbc206bb7099/userdata/shm major:0 minor:433 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d6a85bdb7f1cc7f2e3c02ac1a89a9769dc55dc1bafd34d61098ba3cde6da252/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d6a85bdb7f1cc7f2e3c02ac1a89a9769dc55dc1bafd34d61098ba3cde6da252/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e471c8b707b2bbd6c778f9662b831117839d3f88b59ca306212a3a5091e8077/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e471c8b707b2bbd6c778f9662b831117839d3f88b59ca306212a3a5091e8077/userdata/shm major:0 minor:491 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2f0063099a047671847bd4c92020cb142ff4d070dffaf850669d69aef2b05048/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2f0063099a047671847bd4c92020cb142ff4d070dffaf850669d69aef2b05048/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/30645fbd813ee7e2ef928a485967a08410099870bd752dd7c391d0b10d67ad87/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/30645fbd813ee7e2ef928a485967a08410099870bd752dd7c391d0b10d67ad87/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/350744c898150214cc41dce1779dab592400db1175c630c75b0851a35af5f404/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/350744c898150214cc41dce1779dab592400db1175c630c75b0851a35af5f404/userdata/shm major:0 minor:248 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/35a47295d9f86275eb10f7f578cc2d21fe2637b5c452766ffdbf53d592efa0d2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/35a47295d9f86275eb10f7f578cc2d21fe2637b5c452766ffdbf53d592efa0d2/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/478ad09b369a31d89d2db36a7d36a5cf1414649e3d3f3d53ba3e978d52b8859d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/478ad09b369a31d89d2db36a7d36a5cf1414649e3d3f3d53ba3e978d52b8859d/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/48c537a2e48500aa870420db10b2f9fe6ca3063600d111ce829f3d8342d9f429/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/48c537a2e48500aa870420db10b2f9fe6ca3063600d111ce829f3d8342d9f429/userdata/shm major:0 minor:252 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4cfc28c7646e8a126052a353231f0aab20b4bfe1e31e6b7488dc447085ba219a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4cfc28c7646e8a126052a353231f0aab20b4bfe1e31e6b7488dc447085ba219a/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/542986989c9fe1332f4a001b80ae9f7c3a566c325acb0350072842372c21b519/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/542986989c9fe1332f4a001b80ae9f7c3a566c325acb0350072842372c21b519/userdata/shm major:0 minor:397 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/59fe24e8975ca0ad63c33c5e31b5d30e6cdf24e5497c94ca50e6c25c4015a842/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/59fe24e8975ca0ad63c33c5e31b5d30e6cdf24e5497c94ca50e6c25c4015a842/userdata/shm major:0 minor:489 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f51c29e18d6e587b4eaec9f70a10e1f69be2eb1842d5125c4dae0ef0d221f66/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f51c29e18d6e587b4eaec9f70a10e1f69be2eb1842d5125c4dae0ef0d221f66/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/75a6c1a77d683dc4e2c1c913eb203d9aba20647f8a731e4f4baa0f3a63ebaea6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/75a6c1a77d683dc4e2c1c913eb203d9aba20647f8a731e4f4baa0f3a63ebaea6/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/84fecc8827c4ab0bb51183624b9a312af52113fa78b2052b8e875bf8289eb44a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/84fecc8827c4ab0bb51183624b9a312af52113fa78b2052b8e875bf8289eb44a/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/949e8c265ebd46b758189e30223340b23e1fc52d75d3989e42ba951d4b663b6f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/949e8c265ebd46b758189e30223340b23e1fc52d75d3989e42ba951d4b663b6f/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/975451b57713ff9aaaaf05dc0a6904b669246c28463e3c2012c1bb33393b0376/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/975451b57713ff9aaaaf05dc0a6904b669246c28463e3c2012c1bb33393b0376/userdata/shm major:0 minor:493 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8defe25cc00c430a9110887fea8d6dff2d0a65dfdb922885b8fdf364c92ae62/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8defe25cc00c430a9110887fea8d6dff2d0a65dfdb922885b8fdf364c92ae62/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ba5ce77d099cc1f6928cc92045a63bda220bc2e2fbd0568b148347b236ca589b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ba5ce77d099cc1f6928cc92045a63bda220bc2e2fbd0568b148347b236ca589b/userdata/shm major:0 minor:250 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5ac65ae8ef366a33b346f3b55376013920838985c353f9b7f9b702d5ab7d0ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5ac65ae8ef366a33b346f3b55376013920838985c353f9b7f9b702d5ab7d0ca/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d164a428de2889bd890be21449cde06ccd895263d6ab8392bc534326d6d30224/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d164a428de2889bd890be21449cde06ccd895263d6ab8392bc534326d6d30224/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e158cae34761dcb7e8d465156deae870bc77549e90d3d942895cd9c8fa82ab4a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e158cae34761dcb7e8d465156deae870bc77549e90d3d942895cd9c8fa82ab4a/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ff0bc0fe6e6675667a340512c877189dc978ad51e714f5e355f4cfa516eb9eae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ff0bc0fe6e6675667a340512c877189dc978ad51e714f5e355f4cfa516eb9eae/userdata/shm major:0 minor:319 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84/volumes/kubernetes.io~projected/kube-api-access-wr5ms:{mountpoint:/var/lib/kubelet/pods/1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84/volumes/kubernetes.io~projected/kube-api-access-wr5ms major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:244 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26470ab5-58d3-4b17-bb82-e1b20803e7db/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/26470ab5-58d3-4b17-bb82-e1b20803e7db/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26470ab5-58d3-4b17-bb82-e1b20803e7db/volumes/kubernetes.io~projected/kube-api-access-rpl2j:{mountpoint:/var/lib/kubelet/pods/26470ab5-58d3-4b17-bb82-e1b20803e7db/volumes/kubernetes.io~projected/kube-api-access-rpl2j major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26470ab5-58d3-4b17-bb82-e1b20803e7db/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/26470ab5-58d3-4b17-bb82-e1b20803e7db/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34aecbea-ff97-4f23-8336-326316720a2f/volumes/kubernetes.io~projected/kube-api-access-79nvt:{mountpoint:/var/lib/kubelet/pods/34aecbea-ff97-4f23-8336-326316720a2f/volumes/kubernetes.io~projected/kube-api-access-79nvt major:0 minor:396 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34aecbea-ff97-4f23-8336-326316720a2f/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/34aecbea-ff97-4f23-8336-326316720a2f/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:394 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34aecbea-ff97-4f23-8336-326316720a2f/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/34aecbea-ff97-4f23-8336-326316720a2f/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:395 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/388103ca-b2f5-42b2-96ce-737f1ef00dc1/volumes/kubernetes.io~projected/kube-api-access-sfzkw:{mountpoint:/var/lib/kubelet/pods/388103ca-b2f5-42b2-96ce-737f1ef00dc1/volumes/kubernetes.io~projected/kube-api-access-sfzkw major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5522ff12-fed1-4411-96b0-1cceb6f0b634/volumes/kubernetes.io~projected/kube-api-access-59m4r:{mountpoint:/var/lib/kubelet/pods/5522ff12-fed1-4411-96b0-1cceb6f0b634/volumes/kubernetes.io~projected/kube-api-access-59m4r major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5522ff12-fed1-4411-96b0-1cceb6f0b634/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/5522ff12-fed1-4411-96b0-1cceb6f0b634/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5522ff12-fed1-4411-96b0-1cceb6f0b634/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/5522ff12-fed1-4411-96b0-1cceb6f0b634/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/559e1de2-7e21-4aeb-916f-7a75e0c75841/volumes/kubernetes.io~projected/kube-api-access-bfpdd:{mountpoint:/var/lib/kubelet/pods/559e1de2-7e21-4aeb-916f-7a75e0c75841/volumes/kubernetes.io~projected/kube-api-access-bfpdd major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/559e1de2-7e21-4aeb-916f-7a75e0c75841/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/559e1de2-7e21-4aeb-916f-7a75e0c75841/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/559e1de2-7e21-4aeb-916f-7a75e0c75841/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/559e1de2-7e21-4aeb-916f-7a75e0c75841/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/559e1de2-7e21-4aeb-916f-7a75e0c75841/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/559e1de2-7e21-4aeb-916f-7a75e0c75841/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59e9f4f9-590f-43b0-999d-1fad31779d16/volumes/kubernetes.io~projected/kube-api-access-76469:{mountpoint:/var/lib/kubelet/pods/59e9f4f9-590f-43b0-999d-1fad31779d16/volumes/kubernetes.io~projected/kube-api-access-76469 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59e9f4f9-590f-43b0-999d-1fad31779d16/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/59e9f4f9-590f-43b0-999d-1fad31779d16/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5bcf3164-e6e8-4323-aad0-ee8adbe4a77d/volumes/kubernetes.io~projected/kube-api-access-n5ssp:{mountpoint:/var/lib/kubelet/pods/5bcf3164-e6e8-4323-aad0-ee8adbe4a77d/volumes/kubernetes.io~projected/kube-api-access-n5ssp major:0 minor:358 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5bcf3164-e6e8-4323-aad0-ee8adbe4a77d/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/5bcf3164-e6e8-4323-aad0-ee8adbe4a77d/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:357 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d232021-de4b-4240-8877-e99fe3563101/volumes/kubernetes.io~projected/kube-api-access-qxb5v:{mountpoint:/var/lib/kubelet/pods/5d232021-de4b-4240-8877-e99fe3563101/volumes/kubernetes.io~projected/kube-api-access-qxb5v major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d232021-de4b-4240-8877-e99fe3563101/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/5d232021-de4b-4240-8877-e99fe3563101/volumes/kubernetes.io~secret/metrics-certs major:0 minor:483 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61e0c4ee-edf6-4938-96c0-c79812e00840/volumes/kubernetes.io~projected/kube-api-access-wb9nj:{mountpoint:/var/lib/kubelet/pods/61e0c4ee-edf6-4938-96c0-c79812e00840/volumes/kubernetes.io~projected/kube-api-access-wb9nj major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a/volumes/kubernetes.io~projected/kube-api-access-prd7v:{mountpoint:/var/lib/kubelet/pods/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a/volumes/kubernetes.io~projected/kube-api-access-prd7v major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe/volumes/kubernetes.io~projected/kube-api-access-69mrb:{mountpoint:/var/lib/kubelet/pods/79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe/volumes/kubernetes.io~projected/kube-api-access-69mrb major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9481376a-a63f-46e2-a605-fe4bb019c331/volumes/kubernetes.io~projected/kube-api-access-6d7sj:{mountpoint:/var/lib/kubelet/pods/9481376a-a63f-46e2-a605-fe4bb019c331/volumes/kubernetes.io~projected/kube-api-access-6d7sj major:0 minor:488 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95638663-fffb-41a6-8056-c1f4293a3fb0/volumes/kubernetes.io~projected/kube-api-access-hqzsj:{mountpoint:/var/lib/kubelet/pods/95638663-fffb-41a6-8056-c1f4293a3fb0/volumes/kubernetes.io~projected/kube-api-access-hqzsj major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95638663-fffb-41a6-8056-c1f4293a3fb0/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/95638663-fffb-41a6-8056-c1f4293a3fb0/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:242 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95638663-fffb-41a6-8056-c1f4293a3fb0/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/95638663-fffb-41a6-8056-c1f4293a3fb0/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volume-subpaths/web-config/config-reloader/2 major:0 minor:423 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:423 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volume-subpaths/web-config/prometheus/15 major:0 minor:423 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~empty-dir/config-out major:0 minor:430 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~projected/kube-api-access-95jzn:{mountpoint:/var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~projected/kube-api-access-95jzn major:0 minor:431 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~projected/tls-assets major:0 minor:422 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~secret/config major:0 minor:424 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:425 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:418 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:432 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:429 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:426 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:428 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:427 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~secret/web-config major:0 minor:423 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bac70f0f-4585-42fe-abba-916ac41fcc62/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/bac70f0f-4585-42fe-abba-916ac41fcc62/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:487 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbfd4978-1740-4bf1-b18b-f2d7665fa421/volumes/kubernetes.io~projected/kube-api-access-x756c:{mountpoint:/var/lib/kubelet/pods/bbfd4978-1740-4bf1-b18b-f2d7665fa421/volumes/kubernetes.io~projected/kube-api-access-x756c major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cdcb63f9-d455-4105-8911-33dc1d3dedad/volumes/kubernetes.io~projected/kube-api-access-ld5mg:{mountpoint:/var/lib/kubelet/pods/cdcb63f9-d455-4105-8911-33dc1d3dedad/volumes/kubernetes.io~projected/kube-api-access-ld5mg major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d08dca6d-7563-4d2c-ae92-e7ffd4beda06/volumes/kubernetes.io~projected/kube-api-access-znlq8:{mountpoint:/var/lib/kubelet/pods/d08dca6d-7563-4d2c-ae92-e7ffd4beda06/volumes/kubernetes.io~projected/kube-api-access-znlq8 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d08dca6d-7563-4d2c-ae92-e7ffd4beda06/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/d08dca6d-7563-4d2c-ae92-e7ffd4beda06/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2344442-689b-49f6-a97b-365500025b2d/volumes/kubernetes.io~projected/kube-api-access-wmn8n:{mountpoint:/var/lib/kubelet/pods/d2344442-689b-49f6-a97b-365500025b2d/volumes/kubernetes.io~projected/kube-api-access-wmn8n major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8d094b5-dae6-4655-9425-c10f62f41f07/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/d8d094b5-dae6-4655-9425-c10f62f41f07/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8d094b5-dae6-4655-9425-c10f62f41f07/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/d8d094b5-dae6-4655-9425-c10f62f41f07/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8d094b5-dae6-4655-9425-c10f62f41f07/volumes/kubernetes.io~projected/kube-api-access-b7kw6:{mountpoint:/var/lib/kubelet/pods/d8d094b5-dae6-4655-9425-c10f62f41f07/volumes/kubernetes.io~projected/kube-api-access-b7kw6 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d90c5697-ccdd-4845-8972-dbbd952dcc87/volumes/kubernetes.io~projected/kube-api-access-2zrwn:{mountpoint:/var/lib/kubelet/pods/d90c5697-ccdd-4845-8972-dbbd952dcc87/volumes/kubernetes.io~projected/kube-api-access-2zrwn major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d90c5697-ccdd-4845-8972-dbbd952dcc87/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/d90c5697-ccdd-4845-8972-dbbd952dcc87/volumes/kubernetes.io~secret/cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de6f6762-66c6-42c0-bf0d-7966483a040d/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/de6f6762-66c6-42c0-bf0d-7966483a040d/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volume-subpaths/web-config/alertmanager/9 major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volume-subpaths/web-config/config-reloader/6 major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volumes/kubernetes.io~empty-dir/config-out major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volumes/kubernetes.io~projected/kube-api-access-sprpf:{mountpoint:/var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volumes/kubernetes.io~projected/kube-api-access-sprpf major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volumes/kubernetes.io~projected/tls-assets major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volumes/kubernetes.io~secret/config-volume major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volumes/kubernetes.io~secret/web-config major:0 minor:283 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/7ef0b74014f919410d18619ce143cb4a1f139526be4cffffe52aac0800c051a1/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/2e2aec39e6e7d5d0d4481709bd5708c10856d9b227aef5a989755bb4f49f778d/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/94b242c8dcb0742f15f48abbaaf19755deee6f16679008c91116a1643ec4944c/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/3d751620f517d4123cae90fd30e44a46b90267f24e2ccc4770d63bb627b6015d/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/979535d8a1bbba4ef444ae6663a7b9d5b7ee74b250ac317a18e1cb7db8c2e9ad/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/97414d00b24ab192acafe1b50d3c55259e08c9f9fd9300a97b0870ff3ec8a3a4/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/86e2e14d0b23240ad4ca8347cd81cbc453339cc4bdea183bea1b803ebf905167/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/594242e924180e0e713eee5371a4d8d284440ee9f1738dac4b9463f96535b143/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/15a34fd92c56b1aefdffc400f8683a53d6854bec9128622cad65c8b7f3d317d0/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/c86abbd51c0553f59ca7d8757be5122aed5d98039859ecbd2769a1d38ae1369a/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/a2b82aad99196c34feb9474da193d7bb1c0f4a5f597e241e384c72bc98c9f793/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/3aa4f5a91e1df6d697004c1c8911b00536fa07bb14448d3f922dc3a595e403f6/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/c04f74915e8a7f843fc1b3e00757a34f626c73c838b01c37a0f816773af8a3bf/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/d7bfbba85aa5bc16a9960db45278c3d2c1a9d1659615cc88b3de48396bc3e172/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/d55113f5c3f0395da29bdc7eb35494cb85ac8fde194f3f533696fddd73139be3/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/b47bd96e5f46b376491bee2805c13468ba942f0ba93ca38c7a0fdc5ace305360/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/405c5490d2a08b5caaece398e82a107887daac2613a0cd4afe1027ad2baca2c3/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/c6ae8a122a77741534f8c3273c2e77cb29f23416dfdbb3426111d248fb2fc483/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/9bd6bb3276148c02861bf81c522c4aa5a8876433085bc49b2bdd8d7fc26b1ce2/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/2db7c1feb122b4badf102dce9f9c894587a189576e9798d1279098a1655eff0b/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/f1dca60ccf462601474b4b3b591ac172e00f811672e37a0c97e53cf469c40849/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/f49b728c87ca1d9517f51eb9d8df30ca4c9f89a7a5c89dea305a605ce4286876/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/d8833ffb2faa0d9ea56fe5b55602acc0044a4b617d3399d5d4d1acbeb8458c88/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/a45c2acfe77b0db0a7bccc1e9a852caf565d2393024b38bb25b04c84d5c3e44d/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/23da3b1d67e61462cfbe3273c938b584fb51056d18f718d2f9a99a2eef9abd0f/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/6303e67d16ff8922bcd31f0e5d70e514ee6bc3293cff4e79c9b2f2955e8ac7c3/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/fb88d0f0ce4661f3fecfe197e5cd447d6391f9ae12496d47870d484289c71f20/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/f2877a03d01a32100ef42d2c3ba3aac74caf0091112f60c936b05935d2262590/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/2199840a6fcf3e552e5980147e90035031d1a25d792d575ec24d2e58ac0fbba9/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/a40ade1f95691572d8b4f263428e3bfac357e216d787c1795b63857e071ff48e/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/77b23cab3f84de9f311f1e06d25b1d8969d636ecf112fe207810d8a959ab2511/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/ae4b18e0ef9f5aa1d6c12ce4ccb08613b37f848a66d17c9f109d4cbb958fde3b/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/e6cfeb91e0718946b9bc66a03900228943596fd8e6c7c33329206ff6eaaf66b3/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/3b7d0afe1c3b224c617dedaffe68dcccd18904937641851cc17f9c6e953ceaa0/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/d4bcc46d3c12fa9849937cc0531020a941960c8774ff79b468c54d4c0fb8a7d5/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/980f7c4651c2641caab5640a03c1356b1695f7f3ccee60eb7c19e6de7cb3a4ff/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/2f07fc4d4de7b748310e01989e5be38cb05c02ac983b868fc0cc70d1e8f83889/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/eb902756daba66756cc4c53bdb59496e5dc782a8f6ceeda55abfed00f98482dc/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/eed51b4e2a7a5522a495f4f32515146b5aad0fb9ff151be29a1f8e4d02c69659/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/067a518ceefcfd9f007d917933429be597b5662428dc297683efc0f71c9c8f05/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/b581ac90fc5af596e6baa9599d4da0f01978556776cca7b58a4c93aec6e68b1d/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/e76dd9d89419969629b39bad88b37c1a6460a38eac599fe8dbaa8037c44ebf25/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/a61333241bf39e79677754fca278f6469e5f4df7d801737f035c1f8669070746/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/f3e6bd81cc5a4657e9cfe7c0b087c39b5f78fe0c84cc51e2d82d9d10cad58588/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/dbb8bacadf1bc86d96bf8e07334d15d4c978316ea75cef5cc63279fa563c8889/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/25374cf8618ba9be6086d5795f9879af1319472e0c11d7da58056c634694ab21/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/094e0e4ad80e2a83edd44705961b03bc1fcf0e87384eeb915229a9873f8e217b/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/41e64bac8ec2583898ca36d03fe1a55f1f69ab75c435f84c2b399452100a5141/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/4cdb929b5008cdf7cdaa05cc1cf45f2adce6e5618031da3596a098a96427c936/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/9a38138f894625d6f103817d27646c20493ac13a83ae9cb72a5d6cea4bdaadef/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/1cb5c266023a83e26a18d5b955c38aa4d30175d3268d4680b3a7eda26bfd3ee7/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/3c6afad8b74338904e103dcfa8ca2ce97bd64851a4e3a39f1f247e707bcac825/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/8e6673dc704a1c04f8b3ade0532517ab08a0336f5985437cefdd63536b0773b8/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-399:{mountpoint:/var/lib/containers/storage/overlay/4f90343d4a2dfa4cf96596f791da67a0774579ff958d34fc7198ce6c5ee05c57/merged major:0 minor:399 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/6e6d9da25fef6d7df8f404ec15e8eee0aa157ad4a54ad817c25b8b9d5328e088/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-404:{mountpoint:/var/lib/containers/storage/overlay/64644afd62848a35d620bc8d0a331eae42368501e57cdd3edb4f3a5d207d5905/merged major:0 minor:404 fsType:overlay blockSize:0} overlay_0-409:{mountpoint:/var/lib/containers/storage/overlay/f5cf40d8ce2202f0d41a582bf34872c9303483a74117f4c017831039cd5709a9/merged major:0 minor:409 fsType:overlay blockSize:0} overlay_0-413:{mountpoint:/var/lib/containers/storage/overlay/0b64cdcb0b7f501205de706a77a2842aacb7807e87c9459c21ba35f9161e71ef/merged major:0 minor:413 fsType:overlay blockSize:0} overlay_0-435:{mountpoint:/var/lib/containers/storage/overlay/7d83bc7e7b1c5b2dcdb0ec9497febcc5d338833758f9ee2d6f7964d6cae5bcc9/merged major:0 minor:435 fsType:overlay blockSize:0} overlay_0-437:{mountpoint:/var/lib/containers/storage/overlay/9d4f279b5c3bc040a58478bab53600e9a48d66e47deec1cace13203a1f2ce8d8/merged major:0 minor:437 fsType:overlay blockSize:0} overlay_0-439:{mountpoint:/var/lib/containers/storage/overlay/a5dcee4f923b38852511d91ad48405a775b55f175b071772a920a21fcbb470e8/merged major:0 minor:439 fsType:overlay blockSize:0} overlay_0-441:{mountpoint:/var/lib/containers/storage/overlay/b825c1ff49cb5c0f1292351adc56f45af7f47ca59a84abffe76c53abb5b664a6/merged major:0 minor:441 fsType:overlay blockSize:0} overlay_0-451:{mountpoint:/var/lib/containers/storage/overlay/aeb127e4db37d7552ee9dbcd993a4e3a0ae4d1311dbd3b9c39c22a27b68a1501/merged major:0 minor:451 fsType:overlay blockSize:0} overlay_0-457:{mountpoint:/var/lib/containers/storage/overlay/f7b99c870ce26c19d2f6c2a0d4f12a4616f677f795d39e89b6a0199692aa5a85/merged major:0 minor:457 fsType:overlay blockSize:0} overlay_0-463:{mountpoint:/var/lib/containers/storage/overlay/4ccc74401d48310da80eb443453de7c6a0a970d0dd7cb61ddd4d329bda1cab7e/merged major:0 minor:463 fsType:overlay blockSize:0} overlay_0-468:{mountpoint:/var/lib/containers/storage/overlay/ca32a67ae2d9e1b07a3ce60d795b8c6b9260c8d49dea3115d53ab5e6f8eca33d/merged major:0 minor:468 fsType:overlay blockSize:0} overlay_0-473:{mountpoint:/var/lib/containers/storage/overlay/ac8cc3c6f522cad851bdb8837b4e7ba43dbc12b4739b06dda49e855c18a18c3d/merged major:0 minor:473 fsType:overlay blockSize:0} overlay_0-478:{mountpoint:/var/lib/containers/storage/overlay/720807bbe15c2a73b8fbceb9668964f0eb043fdf1e2017b0efc898a81345de5f/merged major:0 minor:478 fsType:overlay blockSize:0} overlay_0-495:{mountpoint:/var/lib/containers/storage/overlay/58cad05aa559af6ea611b41daa60b694d07245a21cbe9b9c9827c742e4a530b5/merged major:0 minor:495 fsType:overlay blockSize:0} overlay_0-497:{mountpoint:/var/lib/containers/storage/overlay/39209b80ab5a4f175f82be172e86f16007b191842d4c1dc563b81dbef1575ac4/merged major:0 minor:497 fsType:overlay blockSize:0} overlay_0-499:{mountpoint:/var/lib/containers/storage/overlay/7d9a630f05f6242ef9b71da59a2c57168cfc7118842a74ebeab01efcd3ad7198/merged major:0 minor:499 fsType:overlay blockSize:0} overlay_0-501:{mountpoint:/var/lib/containers/storage/overlay/1bac8598b90436815e0e74233aa71793835c2d4936e938ae99ba23ec9e54dddf/merged major:0 minor:501 fsType:overlay blockSize:0} overlay_0-503:{mountpoint:/var/lib/containers/storage/overlay/97516c806d2f6d0f1e98d8589f5d2424f256054bf7cc94d797bd03decd5ed414/merged major:0 minor:503 fsType:overlay blockSize:0} overlay_0-509:{mountpoint:/var/lib/containers/storage/overlay/abb4bc0b301fefd43b0f64444332583821b5241832b91da8f1cbe965ee21872f/merged major:0 minor:509 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/b5ff70fafd260091f57ab433ae7ccfb34f5c995d3990fab4d9dc13a4fd21ec3a/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-514:{mountpoint:/var/lib/containers/storage/overlay/c62bf534f79a1e7f8953919b5ddc8df43080d6f4c9a2adec921bfded56f22338/merged major:0 minor:514 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/297add28a1bcdf88525c8bc75c4636f79b4d33445badc1b4a83c5eafca5d30a3/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/ac34fd6b026c7428257e80077ff6c690b67a188d8cab849116d98d226b41f381/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/2f164789b0f6bb2e607eda4196b3adc378458a5af4e17eccbbb0a1cc6eefa123/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/64c03fef46f5507e20982e0f36c50e0f6c1faa877e001cdfc7728d7ba24b3fab/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/0819c72b7fdc811b788dc7456e0c713db5e27634792fa5df627275a76f8e12bc/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/8ead54a8ba8b1eb7414d18220a3182216e0e4795bf7f54b2bf5e4a71230cbed8/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/f91174c4fb4a96201fa6614d5970e59eacc43f97c6cb2ab33f0f5bd447bb17e0/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/abf81c1f13edeb81e31c94da59238059a84f7c4bf41aff11411e0ad446efe4c4/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/920466f8d9be772e3e309be1f0d374b45afbe3cbc734c98dbffcef3b6a7e75c5/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 04 00:58:35.123096 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.121822 7059 manager.go:217] Machine: {Timestamp:2026-11-04 00:58:35.121004599 +0000 UTC m=+0.062853496 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3058715 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26de6bada58ada926556bddff64dd3 SystemUUID:ec26de6b-ada5-8ada-9265-56bddff64dd3 BootID:1965753a-7ae6-4376-8604-d00adf320a03 Filesystems:[{Device:/var/lib/kubelet/pods/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a/volumes/kubernetes.io~projected/kube-api-access-prd7v DeviceMajor:0 DeviceMinor:314 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0ee3f067670f8fcf04bafdaf8967f414cf98590768a1670fa6221c71f63157d6/userdata/shm DeviceMajor:0 DeviceMinor:387 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:427 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5d232021-de4b-4240-8877-e99fe3563101/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:483 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-499 DeviceMajor:0 DeviceMinor:499 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/de6f6762-66c6-42c0-bf0d-7966483a040d/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:283 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4cfc28c7646e8a126052a353231f0aab20b4bfe1e31e6b7488dc447085ba219a/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5522ff12-fed1-4411-96b0-1cceb6f0b634/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:243 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:307 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/559e1de2-7e21-4aeb-916f-7a75e0c75841/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:335 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:422 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d8d094b5-dae6-4655-9425-c10f62f41f07/volumes/kubernetes.io~projected/kube-api-access-b7kw6 DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/61e0c4ee-edf6-4938-96c0-c79812e00840/volumes/kubernetes.io~projected/kube-api-access-wb9nj DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/34aecbea-ff97-4f23-8336-326316720a2f/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:394 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-409 DeviceMajor:0 DeviceMinor:409 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:424 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9481376a-a63f-46e2-a605-fe4bb019c331/volumes/kubernetes.io~projected/kube-api-access-6d7sj DeviceMajor:0 DeviceMinor:488 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/59fe24e8975ca0ad63c33c5e31b5d30e6cdf24e5497c94ca50e6c25c4015a842/userdata/shm DeviceMajor:0 DeviceMinor:489 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/30645fbd813ee7e2ef928a485967a08410099870bd752dd7c391d0b10d67ad87/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5bcf3164-e6e8-4323-aad0-ee8adbe4a77d/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:357 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:418 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/59e9f4f9-590f-43b0-999d-1fad31779d16/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:423 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-514 DeviceMajor:0 DeviceMinor:514 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/388103ca-b2f5-42b2-96ce-737f1ef00dc1/volumes/kubernetes.io~projected/kube-api-access-sfzkw DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/559e1de2-7e21-4aeb-916f-7a75e0c75841/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:340 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/559e1de2-7e21-4aeb-916f-7a75e0c75841/volumes/kubernetes.io~projected/kube-api-access-bfpdd DeviceMajor:0 DeviceMinor:341 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/34aecbea-ff97-4f23-8336-326316720a2f/volumes/kubernetes.io~projected/kube-api-access-79nvt DeviceMajor:0 DeviceMinor:396 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/542986989c9fe1332f4a001b80ae9f7c3a566c325acb0350072842372c21b519/userdata/shm DeviceMajor:0 DeviceMinor:397 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-463 DeviceMajor:0 DeviceMinor:463 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/975451b57713ff9aaaaf05dc0a6904b669246c28463e3c2012c1bb33393b0376/userdata/shm DeviceMajor:0 DeviceMinor:493 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d90c5697-ccdd-4845-8972-dbbd952dcc87/volumes/kubernetes.io~projected/kube-api-access-2zrwn DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:432 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d2525aeb17bd4389865536271eac885aad37b39e20e47e47da5fbc206bb7099/userdata/shm DeviceMajor:0 DeviceMinor:433 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-435 DeviceMajor:0 DeviceMinor:435 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:283 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e158cae34761dcb7e8d465156deae870bc77549e90d3d942895cd9c8fa82ab4a/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8d094b5-dae6-4655-9425-c10f62f41f07/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:240 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5522ff12-fed1-4411-96b0-1cceb6f0b634/volumes/kubernetes.io~projected/kube-api-access-59m4r DeviceMajor:0 DeviceMinor:245 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:279 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/84fecc8827c4ab0bb51183624b9a312af52113fa78b2052b8e875bf8289eb44a/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5522ff12-fed1-4411-96b0-1cceb6f0b634/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:236 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/48c537a2e48500aa870420db10b2f9fe6ca3063600d111ce829f3d8342d9f429/userdata/shm DeviceMajor:0 DeviceMinor:252 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:278 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/35a47295d9f86275eb10f7f578cc2d21fe2637b5c452766ffdbf53d592efa0d2/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/26470ab5-58d3-4b17-bb82-e1b20803e7db/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/75a6c1a77d683dc4e2c1c913eb203d9aba20647f8a731e4f4baa0f3a63ebaea6/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/288b26a3d92fef361cf864edee46bc80d706a3065e7800ba294dcfc9d2c80a80/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:276 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:283 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~projected/kube-api-access-95jzn DeviceMajor:0 DeviceMinor:431 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-441 DeviceMajor:0 DeviceMinor:441 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-495 DeviceMajor:0 DeviceMinor:495 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-503 DeviceMajor:0 DeviceMinor:503 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cdcb63f9-d455-4105-8911-33dc1d3dedad/volumes/kubernetes.io~projected/kube-api-access-ld5mg DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:426 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:423 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5bcf3164-e6e8-4323-aad0-ee8adbe4a77d/volumes/kubernetes.io~projected/kube-api-access-n5ssp DeviceMajor:0 DeviceMinor:358 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bac70f0f-4585-42fe-abba-916ac41fcc62/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:487 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/949e8c265ebd46b758189e30223340b23e1fc52d75d3989e42ba951d4b663b6f/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5ac65ae8ef366a33b346f3b55376013920838985c353f9b7f9b702d5ab7d0ca/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/59e9f4f9-590f-43b0-999d-1fad31779d16/volumes/kubernetes.io~projected/kube-api-access-76469 DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:428 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-451 DeviceMajor:0 DeviceMinor:451 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:423 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-497 DeviceMajor:0 DeviceMinor:497 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/95638663-fffb-41a6-8056-c1f4293a3fb0/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:242 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:284 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:312 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-439 DeviceMajor:0 DeviceMinor:439 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d2344442-689b-49f6-a97b-365500025b2d/volumes/kubernetes.io~projected/kube-api-access-wmn8n DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d6a85bdb7f1cc7f2e3c02ac1a89a9769dc55dc1bafd34d61098ba3cde6da252/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:423 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e471c8b707b2bbd6c778f9662b831117839d3f88b59ca306212a3a5091e8077/userdata/shm DeviceMajor:0 DeviceMinor:491 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/478ad09b369a31d89d2db36a7d36a5cf1414649e3d3f3d53ba3e978d52b8859d/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d08dca6d-7563-4d2c-ae92-e7ffd4beda06/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:244 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/26470ab5-58d3-4b17-bb82-e1b20803e7db/volumes/kubernetes.io~projected/kube-api-access-rpl2j DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/95638663-fffb-41a6-8056-c1f4293a3fb0/volumes/kubernetes.io~projected/kube-api-access-hqzsj DeviceMajor:0 DeviceMinor:247 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:277 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:284 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-501 DeviceMajor:0 DeviceMinor:501 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe/volumes/kubernetes.io~projected/kube-api-access-69mrb DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d90c5697-ccdd-4845-8972-dbbd952dcc87/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/350744c898150214cc41dce1779dab592400db1175c630c75b0851a35af5f404/userdata/shm DeviceMajor:0 DeviceMinor:248 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8defe25cc00c430a9110887fea8d6dff2d0a65dfdb922885b8fdf364c92ae62/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:308 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:283 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-404 DeviceMajor:0 DeviceMinor:404 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-437 DeviceMajor:0 DeviceMinor:437 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-473 DeviceMajor:0 DeviceMinor:473 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ba5ce77d099cc1f6928cc92045a63bda220bc2e2fbd0568b148347b236ca589b/userdata/shm DeviceMajor:0 DeviceMinor:250 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volumes/kubernetes.io~projected/kube-api-access-sprpf DeviceMajor:0 DeviceMinor:285 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:286 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/26470ab5-58d3-4b17-bb82-e1b20803e7db/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f51c29e18d6e587b4eaec9f70a10e1f69be2eb1842d5125c4dae0ef0d221f66/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-413 DeviceMajor:0 DeviceMinor:413 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-468 DeviceMajor:0 DeviceMinor:468 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-509 DeviceMajor:0 DeviceMinor:509 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/d8d094b5-dae6-4655-9425-c10f62f41f07/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5d232021-de4b-4240-8877-e99fe3563101/volumes/kubernetes.io~projected/kube-api-access-qxb5v DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/95638663-fffb-41a6-8056-c1f4293a3fb0/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:241 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:425 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bbfd4978-1740-4bf1-b18b-f2d7665fa421/volumes/kubernetes.io~projected/kube-api-access-x756c DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84/volumes/kubernetes.io~projected/kube-api-access-wr5ms DeviceMajor:0 DeviceMinor:246 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ff0bc0fe6e6675667a340512c877189dc978ad51e714f5e355f4cfa516eb9eae/userdata/shm DeviceMajor:0 DeviceMinor:319 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-399 DeviceMajor:0 DeviceMinor:399 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-478 DeviceMajor:0 DeviceMinor:478 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:429 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d164a428de2889bd890be21449cde06ccd895263d6ab8392bc534326d6d30224/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d08dca6d-7563-4d2c-ae92-e7ffd4beda06/volumes/kubernetes.io~projected/kube-api-access-znlq8 DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:306 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:313 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:430 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-457 DeviceMajor:0 DeviceMinor:457 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:305 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/559e1de2-7e21-4aeb-916f-7a75e0c75841/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:339 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/34aecbea-ff97-4f23-8336-326316720a2f/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:395 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2f0063099a047671847bd4c92020cb142ff4d070dffaf850669d69aef2b05048/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:274 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:275 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0ee3f067670f8fc MacAddress:be:e6:1e:68:6a:bd Speed:10000 Mtu:8901} {Name:288b26a3d92fef3 MacAddress:1e:4f:f5:bd:58:31 Speed:10000 Mtu:8901} {Name:2d2525aeb17bd43 MacAddress:36:4e:2a:25:3e:4e Speed:10000 Mtu:8901} {Name:2d6a85bdb7f1cc7 MacAddress:ea:58:4c:d6:5e:c9 Speed:10000 Mtu:8901} {Name:2e471c8b707b2bb MacAddress:4a:29:d4:38:61:08 Speed:10000 Mtu:8901} {Name:350744c89815021 MacAddress:02:0f:3c:57:d1:05 Speed:10000 Mtu:8901} {Name:542986989c9fe13 MacAddress:1e:69:7d:bd:ba:ea Speed:10000 Mtu:8901} {Name:59fe24e8975ca0a MacAddress:a6:3a:e4:6e:3e:f6 Speed:10000 Mtu:8901} {Name:5f51c29e18d6e58 MacAddress:9e:b1:6e:b0:d9:d9 Speed:10000 Mtu:8901} {Name:84fecc8827c4ab0 MacAddress:b2:28:86:5a:62:11 Speed:10000 Mtu:8901} {Name:975451b57713ff9 MacAddress:f2:30:8c:71:3f:0c Speed:10000 Mtu:8901} {Name:a8defe25cc00c43 MacAddress:ae:09:36:98:95:7c Speed:10000 Mtu:8901} {Name:ba5ce77d099cc1f MacAddress:92:e1:64:34:3d:e5 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d4:5b:94:15 Speed:0 Mtu:9001} {Name:br-int MacAddress:82:68:bd:96:96:df Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:d4:5b:94:15 Speed:0 Mtu:9001} {Name:ff0bc0fe6e66756 MacAddress:b2:7e:06:69:5b:ac Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:e2:9c:11:58:09:c5 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:e2:07:01:21:4e:83 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 04 00:58:35.123096 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.122765 7059 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 04 00:58:35.123096 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.122827 7059 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 04 00:58:35.123577 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.123181 7059 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 04 00:58:35.123577 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.123205 7059 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-129.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 04 00:58:35.123577 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.123399 7059 topology_manager.go:143] "Creating topology manager with none policy" Nov 04 00:58:35.123577 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.123410 7059 container_manager_linux.go:311] "Creating device plugin manager" Nov 04 00:58:35.123577 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.123429 7059 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 00:58:35.123577 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.123460 7059 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 00:58:35.123577 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.123571 7059 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 04 00:58:35.123914 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.123781 7059 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 04 00:58:35.123997 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.123940 7059 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 04 00:58:35.123997 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.123985 7059 kubelet.go:493] "Attempting to sync node with API server" Nov 04 00:58:35.124083 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.124000 7059 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 04 00:58:35.124083 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.124014 7059 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 04 00:58:35.124083 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.124027 7059 kubelet.go:405] "Adding apiserver pod source" Nov 04 00:58:35.124083 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.124038 7059 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 04 00:58:35.125265 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.125244 7059 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 04 00:58:35.125265 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.125264 7059 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 00:58:35.125770 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.125749 7059 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 04 00:58:35.128969 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.128954 7059 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 04 00:58:35.128969 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.128968 7059 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 00:58:35.129068 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.129059 7059 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 04 00:58:35.129254 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.129239 7059 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 04 00:58:35.129326 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.129273 7059 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 04 00:58:35.129365 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.129356 7059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 04 00:58:35.129395 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.129372 7059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 04 00:58:35.129395 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.129379 7059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 04 00:58:35.129395 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.129388 7059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 04 00:58:35.129477 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.129396 7059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 04 00:58:35.129477 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.129402 7059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 04 00:58:35.129477 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.129408 7059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 04 00:58:35.129477 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.129414 7059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 04 00:58:35.129477 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.129420 7059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 04 00:58:35.129477 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.129425 7059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 04 00:58:35.129477 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.129438 7059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 04 00:58:35.129477 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.129447 7059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 04 00:58:35.129477 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.129473 7059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 04 00:58:35.129711 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.129482 7059 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 04 00:58:35.133059 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.133043 7059 server.go:1263] "Started kubelet" Nov 04 00:58:35.133200 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.133145 7059 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 04 00:58:35.133591 ip-10-0-128-129 systemd[1]: Started Kubernetes Kubelet. Nov 04 00:58:35.134363 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.133179 7059 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 04 00:58:35.134363 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.133898 7059 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 04 00:58:35.136007 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.135983 7059 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 04 00:58:35.138786 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.138068 7059 server.go:317] "Adding debug handlers to kubelet server" Nov 04 00:58:35.141381 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.141361 7059 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 04 00:58:35.141628 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.141424 7059 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 04 00:58:35.141628 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.141477 7059 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 00:52:25 +0000 UTC" deadline="2028-03-29 15:50:20.237899482 +0000 UTC" Nov 04 00:58:35.141628 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.141492 7059 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12278h51m45.096409679s" Nov 04 00:58:35.141628 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.141557 7059 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 04 00:58:35.141628 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.141570 7059 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 04 00:58:35.141884 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.141668 7059 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 04 00:58:35.143323 ip-10-0-128-129 kubenswrapper[7059]: E1104 00:58:35.142482 7059 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-129.us-east-2.compute.internal\" not found" Nov 04 00:58:35.143323 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.142593 7059 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:58:35.143991 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.143862 7059 factory.go:55] Registering systemd factory Nov 04 00:58:35.143991 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.143881 7059 factory.go:223] Registration of the systemd container factory successfully Nov 04 00:58:35.144670 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.144649 7059 factory.go:153] Registering CRI-O factory Nov 04 00:58:35.144670 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.144671 7059 factory.go:223] Registration of the crio container factory successfully Nov 04 00:58:35.144823 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.144734 7059 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 04 00:58:35.144823 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.144755 7059 factory.go:103] Registering Raw factory Nov 04 00:58:35.144823 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.144769 7059 manager.go:1199] Started watching for new ooms in manager Nov 04 00:58:35.146218 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.145119 7059 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:58:35.146218 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.145129 7059 manager.go:319] Starting recovery of all containers Nov 04 00:58:35.146218 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.145288 7059 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.147336 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" volumeName="kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.147394 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" volumeName="kubernetes.io/empty-dir/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-config-out" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.147411 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d08dca6d-7563-4d2c-ae92-e7ffd4beda06" volumeName="kubernetes.io/empty-dir/d08dca6d-7563-4d2c-ae92-e7ffd4beda06-tmp-dir" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.147438 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" volumeName="kubernetes.io/projected/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-tls-assets" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.147547 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59e9f4f9-590f-43b0-999d-1fad31779d16" volumeName="kubernetes.io/empty-dir/59e9f4f9-590f-43b0-999d-1fad31779d16-data-volume" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.147561 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95638663-fffb-41a6-8056-c1f4293a3fb0" volumeName="kubernetes.io/projected/95638663-fffb-41a6-8056-c1f4293a3fb0-kube-api-access-hqzsj" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.147592 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95638663-fffb-41a6-8056-c1f4293a3fb0" volumeName="kubernetes.io/secret/95638663-fffb-41a6-8056-c1f4293a3fb0-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.147607 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbfd4978-1740-4bf1-b18b-f2d7665fa421" volumeName="kubernetes.io/configmap/bbfd4978-1740-4bf1-b18b-f2d7665fa421-cni-sysctl-allowlist" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.147744 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="559e1de2-7e21-4aeb-916f-7a75e0c75841" volumeName="kubernetes.io/empty-dir/559e1de2-7e21-4aeb-916f-7a75e0c75841-audit-log" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.147781 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" volumeName="kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-metrics-client-certs" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.147800 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de6f6762-66c6-42c0-bf0d-7966483a040d" volumeName="kubernetes.io/configmap/de6f6762-66c6-42c0-bf0d-7966483a040d-konnectivity-ca" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.147824 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34aecbea-ff97-4f23-8336-326316720a2f" volumeName="kubernetes.io/secret/34aecbea-ff97-4f23-8336-326316720a2f-console-serving-cert" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.147852 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5522ff12-fed1-4411-96b0-1cceb6f0b634" volumeName="kubernetes.io/configmap/5522ff12-fed1-4411-96b0-1cceb6f0b634-metrics-client-ca" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.147869 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" volumeName="kubernetes.io/projected/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-tls-assets" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.147886 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" volumeName="kubernetes.io/empty-dir/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-config-out" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.147914 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5522ff12-fed1-4411-96b0-1cceb6f0b634" volumeName="kubernetes.io/empty-dir/5522ff12-fed1-4411-96b0-1cceb6f0b634-node-exporter-textfile" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.147944 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d08dca6d-7563-4d2c-ae92-e7ffd4beda06" volumeName="kubernetes.io/secret/d08dca6d-7563-4d2c-ae92-e7ffd4beda06-metrics-tls" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.147957 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de6f6762-66c6-42c0-bf0d-7966483a040d" volumeName="kubernetes.io/secret/de6f6762-66c6-42c0-bf0d-7966483a040d-agent-certs" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.147990 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" volumeName="kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-main-tls" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.148004 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95638663-fffb-41a6-8056-c1f4293a3fb0" volumeName="kubernetes.io/configmap/95638663-fffb-41a6-8056-c1f4293a3fb0-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.148022 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="559e1de2-7e21-4aeb-916f-7a75e0c75841" volumeName="kubernetes.io/secret/559e1de2-7e21-4aeb-916f-7a75e0c75841-client-ca-bundle" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.148123 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d232021-de4b-4240-8877-e99fe3563101" volumeName="kubernetes.io/secret/5d232021-de4b-4240-8877-e99fe3563101-metrics-certs" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.148137 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" volumeName="kubernetes.io/projected/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-kube-api-access-sprpf" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.148168 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34aecbea-ff97-4f23-8336-326316720a2f" volumeName="kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-oauth-serving-cert" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.148192 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a" volumeName="kubernetes.io/configmap/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-metrics-client-ca" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.148224 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a" volumeName="kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.148277 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95638663-fffb-41a6-8056-c1f4293a3fb0" volumeName="kubernetes.io/empty-dir/95638663-fffb-41a6-8056-c1f4293a3fb0-volume-directive-shadow" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.148311 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" volumeName="kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-web-config" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.148355 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84" volumeName="kubernetes.io/secret/1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84-openshift-state-metrics-tls" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.148393 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26470ab5-58d3-4b17-bb82-e1b20803e7db" volumeName="kubernetes.io/configmap/26470ab5-58d3-4b17-bb82-e1b20803e7db-env-overrides" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.148424 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="559e1de2-7e21-4aeb-916f-7a75e0c75841" volumeName="kubernetes.io/configmap/559e1de2-7e21-4aeb-916f-7a75e0c75841-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.148455 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe" volumeName="kubernetes.io/projected/79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe-kube-api-access-69mrb" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.148467 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" volumeName="kubernetes.io/projected/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-kube-api-access-95jzn" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.148497 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" volumeName="kubernetes.io/empty-dir/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-alertmanager-main-db" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.148534 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34aecbea-ff97-4f23-8336-326316720a2f" volumeName="kubernetes.io/projected/34aecbea-ff97-4f23-8336-326316720a2f-kube-api-access-79nvt" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.148546 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5522ff12-fed1-4411-96b0-1cceb6f0b634" volumeName="kubernetes.io/configmap/5522ff12-fed1-4411-96b0-1cceb6f0b634-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.148579 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a" volumeName="kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-grpc-tls" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.148592 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" volumeName="kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-config-volume" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.148659 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" volumeName="kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.148757 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="388103ca-b2f5-42b2-96ce-737f1ef00dc1" volumeName="kubernetes.io/empty-dir/388103ca-b2f5-42b2-96ce-737f1ef00dc1-tmp-dir" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.148792 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d08dca6d-7563-4d2c-ae92-e7ffd4beda06" volumeName="kubernetes.io/projected/d08dca6d-7563-4d2c-ae92-e7ffd4beda06-kube-api-access-znlq8" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.148809 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" volumeName="kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-web-config" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.148851 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26470ab5-58d3-4b17-bb82-e1b20803e7db" volumeName="kubernetes.io/projected/26470ab5-58d3-4b17-bb82-e1b20803e7db-kube-api-access-rpl2j" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.148864 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" volumeName="kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.148881 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" volumeName="kubernetes.io/empty-dir/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-db" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.148954 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" volumeName="kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-cluster-tls-config" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.148973 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5522ff12-fed1-4411-96b0-1cceb6f0b634" volumeName="kubernetes.io/secret/5522ff12-fed1-4411-96b0-1cceb6f0b634-node-exporter-tls" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.148997 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a" volumeName="kubernetes.io/empty-dir/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-tmp" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.149023 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d90c5697-ccdd-4845-8972-dbbd952dcc87" volumeName="kubernetes.io/projected/d90c5697-ccdd-4845-8972-dbbd952dcc87-kube-api-access-2zrwn" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.149042 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9481376a-a63f-46e2-a605-fe4bb019c331" volumeName="kubernetes.io/projected/9481376a-a63f-46e2-a605-fe4bb019c331-kube-api-access-6d7sj" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.149054 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="559e1de2-7e21-4aeb-916f-7a75e0c75841" volumeName="kubernetes.io/projected/559e1de2-7e21-4aeb-916f-7a75e0c75841-kube-api-access-bfpdd" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.149128 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8d094b5-dae6-4655-9425-c10f62f41f07" volumeName="kubernetes.io/projected/d8d094b5-dae6-4655-9425-c10f62f41f07-kube-api-access-b7kw6" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.149176 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" volumeName="kubernetes.io/configmap/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-metrics-client-ca" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.149239 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d232021-de4b-4240-8877-e99fe3563101" volumeName="kubernetes.io/projected/5d232021-de4b-4240-8877-e99fe3563101-kube-api-access-qxb5v" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.149298 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="388103ca-b2f5-42b2-96ce-737f1ef00dc1" volumeName="kubernetes.io/projected/388103ca-b2f5-42b2-96ce-737f1ef00dc1-kube-api-access-sfzkw" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.149328 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" volumeName="kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-trusted-ca-bundle" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.149411 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbfd4978-1740-4bf1-b18b-f2d7665fa421" volumeName="kubernetes.io/projected/bbfd4978-1740-4bf1-b18b-f2d7665fa421-kube-api-access-x756c" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.149445 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8d094b5-dae6-4655-9425-c10f62f41f07" volumeName="kubernetes.io/empty-dir/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-tuned" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.149458 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5522ff12-fed1-4411-96b0-1cceb6f0b634" volumeName="kubernetes.io/projected/5522ff12-fed1-4411-96b0-1cceb6f0b634-kube-api-access-59m4r" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.149808 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95638663-fffb-41a6-8056-c1f4293a3fb0" volumeName="kubernetes.io/configmap/95638663-fffb-41a6-8056-c1f4293a3fb0-metrics-client-ca" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.149871 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe" volumeName="kubernetes.io/configmap/79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe-iptables-alerter-script" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.149904 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" volumeName="kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-configmap-metrics-client-ca" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.149935 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84" volumeName="kubernetes.io/configmap/1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84-metrics-client-ca" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.149968 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" volumeName="kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.149985 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" volumeName="kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-thanos-prometheus-http-client-file" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.149999 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" volumeName="kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.150031 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" volumeName="kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.150044 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" volumeName="kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-grpc-tls" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.150058 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" volumeName="kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-prometheus-k8s-tls" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.150092 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84" volumeName="kubernetes.io/projected/1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84-kube-api-access-wr5ms" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.150110 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34aecbea-ff97-4f23-8336-326316720a2f" volumeName="kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-service-ca" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.150138 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59e9f4f9-590f-43b0-999d-1fad31779d16" volumeName="kubernetes.io/configmap/59e9f4f9-590f-43b0-999d-1fad31779d16-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.150163 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" volumeName="kubernetes.io/configmap/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.150179 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5522ff12-fed1-4411-96b0-1cceb6f0b634" volumeName="kubernetes.io/secret/5522ff12-fed1-4411-96b0-1cceb6f0b634-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.150195 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bcf3164-e6e8-4323-aad0-ee8adbe4a77d" volumeName="kubernetes.io/secret/5bcf3164-e6e8-4323-aad0-ee8adbe4a77d-monitoring-plugin-cert" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.150233 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbfd4978-1740-4bf1-b18b-f2d7665fa421" volumeName="kubernetes.io/configmap/bbfd4978-1740-4bf1-b18b-f2d7665fa421-cni-binary-copy" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.150258 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cdcb63f9-d455-4105-8911-33dc1d3dedad" volumeName="kubernetes.io/projected/cdcb63f9-d455-4105-8911-33dc1d3dedad-kube-api-access-ld5mg" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.150291 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26470ab5-58d3-4b17-bb82-e1b20803e7db" volumeName="kubernetes.io/configmap/26470ab5-58d3-4b17-bb82-e1b20803e7db-ovnkube-config" seLinuxMountContext="" Nov 04 00:58:35.150274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.150313 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34aecbea-ff97-4f23-8336-326316720a2f" volumeName="kubernetes.io/secret/34aecbea-ff97-4f23-8336-326316720a2f-console-oauth-config" seLinuxMountContext="" Nov 04 00:58:35.153253 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.150402 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59e9f4f9-590f-43b0-999d-1fad31779d16" volumeName="kubernetes.io/projected/59e9f4f9-590f-43b0-999d-1fad31779d16-kube-api-access-76469" seLinuxMountContext="" Nov 04 00:58:35.153253 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.150439 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95638663-fffb-41a6-8056-c1f4293a3fb0" volumeName="kubernetes.io/secret/95638663-fffb-41a6-8056-c1f4293a3fb0-kube-state-metrics-tls" seLinuxMountContext="" Nov 04 00:58:35.153253 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.150470 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26470ab5-58d3-4b17-bb82-e1b20803e7db" volumeName="kubernetes.io/secret/26470ab5-58d3-4b17-bb82-e1b20803e7db-ovn-node-metrics-cert" seLinuxMountContext="" Nov 04 00:58:35.153253 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.150487 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34aecbea-ff97-4f23-8336-326316720a2f" volumeName="kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-console-config" seLinuxMountContext="" Nov 04 00:58:35.153253 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.150524 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a" volumeName="kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Nov 04 00:58:35.153253 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.150593 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" volumeName="kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Nov 04 00:58:35.153253 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.151007 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59e9f4f9-590f-43b0-999d-1fad31779d16" volumeName="kubernetes.io/secret/59e9f4f9-590f-43b0-999d-1fad31779d16-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 04 00:58:35.153253 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.151063 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" volumeName="kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Nov 04 00:58:35.153253 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.151078 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d08dca6d-7563-4d2c-ae92-e7ffd4beda06" volumeName="kubernetes.io/configmap/d08dca6d-7563-4d2c-ae92-e7ffd4beda06-config-volume" seLinuxMountContext="" Nov 04 00:58:35.153253 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.151134 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d90c5697-ccdd-4845-8972-dbbd952dcc87" volumeName="kubernetes.io/secret/d90c5697-ccdd-4845-8972-dbbd952dcc87-cert" seLinuxMountContext="" Nov 04 00:58:35.153253 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.151160 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="559e1de2-7e21-4aeb-916f-7a75e0c75841" volumeName="kubernetes.io/configmap/559e1de2-7e21-4aeb-916f-7a75e0c75841-metrics-server-audit-profiles" seLinuxMountContext="" Nov 04 00:58:35.153253 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.152404 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61e0c4ee-edf6-4938-96c0-c79812e00840" volumeName="kubernetes.io/configmap/61e0c4ee-edf6-4938-96c0-c79812e00840-serviceca" seLinuxMountContext="" Nov 04 00:58:35.153253 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.152678 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a" volumeName="kubernetes.io/projected/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-kube-api-access-prd7v" seLinuxMountContext="" Nov 04 00:58:35.153253 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.152697 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34aecbea-ff97-4f23-8336-326316720a2f" volumeName="kubernetes.io/empty-dir/34aecbea-ff97-4f23-8336-326316720a2f-tmp" seLinuxMountContext="" Nov 04 00:58:35.153253 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.152780 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bcf3164-e6e8-4323-aad0-ee8adbe4a77d" volumeName="kubernetes.io/projected/5bcf3164-e6e8-4323-aad0-ee8adbe4a77d-kube-api-access-n5ssp" seLinuxMountContext="" Nov 04 00:58:35.153253 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.152800 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a" volumeName="kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Nov 04 00:58:35.153253 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.152968 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cdcb63f9-d455-4105-8911-33dc1d3dedad" volumeName="kubernetes.io/configmap/cdcb63f9-d455-4105-8911-33dc1d3dedad-multus-daemon-config" seLinuxMountContext="" Nov 04 00:58:35.153253 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.153228 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" volumeName="kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Nov 04 00:58:35.153876 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.153392 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26470ab5-58d3-4b17-bb82-e1b20803e7db" volumeName="kubernetes.io/configmap/26470ab5-58d3-4b17-bb82-e1b20803e7db-ovnkube-script-lib" seLinuxMountContext="" Nov 04 00:58:35.153876 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.153437 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="559e1de2-7e21-4aeb-916f-7a75e0c75841" volumeName="kubernetes.io/secret/559e1de2-7e21-4aeb-916f-7a75e0c75841-secret-metrics-server-tls" seLinuxMountContext="" Nov 04 00:58:35.153876 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.153475 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a" volumeName="kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Nov 04 00:58:35.153876 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.153515 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" volumeName="kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Nov 04 00:58:35.153876 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.153538 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cdcb63f9-d455-4105-8911-33dc1d3dedad" volumeName="kubernetes.io/configmap/cdcb63f9-d455-4105-8911-33dc1d3dedad-cni-binary-copy" seLinuxMountContext="" Nov 04 00:58:35.154071 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.153905 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a" volumeName="kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-thanos-querier-tls" seLinuxMountContext="" Nov 04 00:58:35.154071 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.153956 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84" volumeName="kubernetes.io/secret/1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 04 00:58:35.154071 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.153997 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="559e1de2-7e21-4aeb-916f-7a75e0c75841" volumeName="kubernetes.io/secret/559e1de2-7e21-4aeb-916f-7a75e0c75841-secret-metrics-server-client-certs" seLinuxMountContext="" Nov 04 00:58:35.154071 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.154015 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bac70f0f-4585-42fe-abba-916ac41fcc62" volumeName="kubernetes.io/secret/bac70f0f-4585-42fe-abba-916ac41fcc62-original-pull-secret" seLinuxMountContext="" Nov 04 00:58:35.154071 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.154061 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbfd4978-1740-4bf1-b18b-f2d7665fa421" volumeName="kubernetes.io/configmap/bbfd4978-1740-4bf1-b18b-f2d7665fa421-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 04 00:58:35.154511 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.154487 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2344442-689b-49f6-a97b-365500025b2d" volumeName="kubernetes.io/projected/d2344442-689b-49f6-a97b-365500025b2d-kube-api-access-wmn8n" seLinuxMountContext="" Nov 04 00:58:35.154572 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.154533 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34aecbea-ff97-4f23-8336-326316720a2f" volumeName="kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-trusted-ca-bundle" seLinuxMountContext="" Nov 04 00:58:35.154572 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.154549 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" volumeName="kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-config" seLinuxMountContext="" Nov 04 00:58:35.154687 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.154563 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" volumeName="kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-kube-rbac-proxy" seLinuxMountContext="" Nov 04 00:58:35.154687 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.154636 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8d094b5-dae6-4655-9425-c10f62f41f07" volumeName="kubernetes.io/empty-dir/d8d094b5-dae6-4655-9425-c10f62f41f07-tmp" seLinuxMountContext="" Nov 04 00:58:35.154780 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.154715 7059 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61e0c4ee-edf6-4938-96c0-c79812e00840" volumeName="kubernetes.io/projected/61e0c4ee-edf6-4938-96c0-c79812e00840-kube-api-access-wb9nj" seLinuxMountContext="" Nov 04 00:58:35.154780 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.154729 7059 reconstruct.go:97] "Volume reconstruction finished" Nov 04 00:58:35.154780 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.154737 7059 reconciler.go:29] "Reconciler: start to sync state" Nov 04 00:58:35.158955 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.158893 7059 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 04 00:58:35.167378 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.167334 7059 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 04 00:58:35.169295 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.169276 7059 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 04 00:58:35.169378 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.169297 7059 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 04 00:58:35.169378 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.169346 7059 kubelet.go:2521] "Starting kubelet main sync loop" Nov 04 00:58:35.169494 ip-10-0-128-129 kubenswrapper[7059]: E1104 00:58:35.169400 7059 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 04 00:58:35.172694 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.172674 7059 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:58:35.179367 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.179316 7059 generic.go:356] "Generic (PLEG): container finished" podID="12bcfb3a1cdb54b757405d90e15ad22f" containerID="9a42e94fa931a44e3e8fbe94eeadb6301397796cfbba9808d5bd07814a38e38f" exitCode=0 Nov 04 00:58:35.214653 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.214624 7059 generic.go:356] "Generic (PLEG): container finished" podID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerID="223d2944c74314599510a4f7c0349335629caf4e673f355fc2d608fd232009d7" exitCode=0 Nov 04 00:58:35.216871 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.216843 7059 generic.go:356] "Generic (PLEG): container finished" podID="5522ff12-fed1-4411-96b0-1cceb6f0b634" containerID="5cada670732f2ccab326c7baf7642add2f207334e3b96ee66f50a2f68552a493" exitCode=0 Nov 04 00:58:35.227160 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.227127 7059 generic.go:356] "Generic (PLEG): container finished" podID="bbfd4978-1740-4bf1-b18b-f2d7665fa421" containerID="01fa48d2462f5916942ddb588d8e0e3bd19922228f64799346e44531e2e9e20d" exitCode=0 Nov 04 00:58:35.227244 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.227164 7059 generic.go:356] "Generic (PLEG): container finished" podID="bbfd4978-1740-4bf1-b18b-f2d7665fa421" containerID="ebb05f5e443426872a507070cb1c56b879882b49d79a07c47cf523ea91c11e77" exitCode=0 Nov 04 00:58:35.227244 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.227175 7059 generic.go:356] "Generic (PLEG): container finished" podID="bbfd4978-1740-4bf1-b18b-f2d7665fa421" containerID="7b744fe485874da7043a4f229d29b1f2ea6cc8abeb79da7a46a0fa496d59a53c" exitCode=0 Nov 04 00:58:35.227244 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.227185 7059 generic.go:356] "Generic (PLEG): container finished" podID="bbfd4978-1740-4bf1-b18b-f2d7665fa421" containerID="7f90d5080dc5449fc083795d40bb782ef8437149ef300238b3ad23509951372d" exitCode=0 Nov 04 00:58:35.227244 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.227194 7059 generic.go:356] "Generic (PLEG): container finished" podID="bbfd4978-1740-4bf1-b18b-f2d7665fa421" containerID="fa60e09e2c8c27515527f9822c7f7171958f7956bbec706a7fa21a64e7139845" exitCode=0 Nov 04 00:58:35.227244 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.227202 7059 generic.go:356] "Generic (PLEG): container finished" podID="bbfd4978-1740-4bf1-b18b-f2d7665fa421" containerID="e2ecb5bc0c15e81b9995dc5b0ff7fd6c2169588fec8f4c3688b1eea77c7fd820" exitCode=0 Nov 04 00:58:35.234008 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.233951 7059 generic.go:356] "Generic (PLEG): container finished" podID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerID="1449e5c536ddfc17d9303f1dc08eb2d596ecca01af40809b528ae00d0b6f96e1" exitCode=0 Nov 04 00:58:35.269485 ip-10-0-128-129 kubenswrapper[7059]: E1104 00:58:35.269463 7059 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 04 00:58:35.271261 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.271245 7059 manager.go:324] Recovery completed Nov 04 00:58:35.295681 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.295660 7059 cpu_manager.go:226] "Starting" policy="none" Nov 04 00:58:35.295681 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.295679 7059 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 04 00:58:35.295799 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.295697 7059 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 04 00:58:35.295870 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.295853 7059 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 04 00:58:35.295900 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.295873 7059 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 04 00:58:35.295900 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.295895 7059 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 04 00:58:35.295993 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.295903 7059 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 04 00:58:35.295993 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.295911 7059 policy_none.go:50] "Start" Nov 04 00:58:35.295993 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.295948 7059 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 04 00:58:35.295993 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.295960 7059 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 04 00:58:35.296099 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.296089 7059 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 04 00:58:35.296099 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.296096 7059 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 04 00:58:35.296156 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.296102 7059 policy_none.go:44] "Start" Nov 04 00:58:35.300778 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.300750 7059 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 04 00:58:35.305389 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.305376 7059 manager.go:341] "Starting Device Plugin manager" Nov 04 00:58:35.305832 ip-10-0-128-129 kubenswrapper[7059]: E1104 00:58:35.305400 7059 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 04 00:58:35.305832 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.305833 7059 server.go:87] "Starting device plugin registration server" Nov 04 00:58:35.306130 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.306117 7059 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 04 00:58:35.306197 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.306129 7059 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 04 00:58:35.306245 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.306203 7059 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 04 00:58:35.306319 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.306294 7059 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 04 00:58:35.306363 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.306326 7059 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 04 00:58:35.306363 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.306330 7059 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 04 00:58:35.306453 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.306373 7059 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-04T00:58:35.306317904Z","UUID":"c4721309-ff03-41b1-95be-f6099295ec56","Handler":null,"Name":"","Endpoint":""} Nov 04 00:58:35.307767 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.307739 7059 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 04 00:58:35.307886 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.307774 7059 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 04 00:58:35.308103 ip-10-0-128-129 kubenswrapper[7059]: E1104 00:58:35.308082 7059 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 04 00:58:35.406707 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.406681 7059 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:58:35.408740 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.408722 7059 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-129.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:58:35.408828 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.408753 7059 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-129.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:58:35.408828 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.408763 7059 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-129.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:58:35.408828 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.408806 7059 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:58:35.420865 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.420847 7059 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:58:35.420919 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.420899 7059 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:58:35.470456 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.470417 7059 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-129.us-east-2.compute.internal"] Nov 04 00:58:35.471007 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.470977 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal" event={"ID":"12bcfb3a1cdb54b757405d90e15ad22f","Type":"ContainerStarted","Data":"35a47295d9f86275eb10f7f578cc2d21fe2637b5c452766ffdbf53d592efa0d2"} Nov 04 00:58:35.471077 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.471014 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-129.us-east-2.compute.internal" event={"ID":"14ca05fc22f96fd5090f819c755a4a44","Type":"ContainerStarted","Data":"d90e5e959fd335473c6b6f60266bdeb6f5524bfbc55841555fedec71947b1054"} Nov 04 00:58:35.471077 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.471028 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-129.us-east-2.compute.internal" event={"ID":"14ca05fc22f96fd5090f819c755a4a44","Type":"ContainerStarted","Data":"e158cae34761dcb7e8d465156deae870bc77549e90d3d942895cd9c8fa82ab4a"} Nov 04 00:58:35.471148 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.471097 7059 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:58:35.471234 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.471220 7059 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:58:35.484975 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.484918 7059 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 00:58:35.484975 ip-10-0-128-129 kubenswrapper[7059]: E1104 00:58:35.484972 7059 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-129.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:58:35.485490 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.485476 7059 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 00:58:35.485541 ip-10-0-128-129 kubenswrapper[7059]: E1104 00:58:35.485511 7059 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:58:35.560812 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.560787 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/14ca05fc22f96fd5090f819c755a4a44-config\") pod \"kube-apiserver-proxy-ip-10-0-128-129.us-east-2.compute.internal\" (UID: \"14ca05fc22f96fd5090f819c755a4a44\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:58:35.560889 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.560816 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/12bcfb3a1cdb54b757405d90e15ad22f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal\" (UID: \"12bcfb3a1cdb54b757405d90e15ad22f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:58:35.560889 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.560833 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12bcfb3a1cdb54b757405d90e15ad22f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal\" (UID: \"12bcfb3a1cdb54b757405d90e15ad22f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:58:35.661399 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.661375 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/14ca05fc22f96fd5090f819c755a4a44-config\") pod \"kube-apiserver-proxy-ip-10-0-128-129.us-east-2.compute.internal\" (UID: \"14ca05fc22f96fd5090f819c755a4a44\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:58:35.661477 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.661403 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/12bcfb3a1cdb54b757405d90e15ad22f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal\" (UID: \"12bcfb3a1cdb54b757405d90e15ad22f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:58:35.661477 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.661420 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12bcfb3a1cdb54b757405d90e15ad22f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal\" (UID: \"12bcfb3a1cdb54b757405d90e15ad22f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:58:35.661477 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.661466 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12bcfb3a1cdb54b757405d90e15ad22f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal\" (UID: \"12bcfb3a1cdb54b757405d90e15ad22f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:58:35.661580 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.661481 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/14ca05fc22f96fd5090f819c755a4a44-config\") pod \"kube-apiserver-proxy-ip-10-0-128-129.us-east-2.compute.internal\" (UID: \"14ca05fc22f96fd5090f819c755a4a44\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:58:35.661580 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:35.661496 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/12bcfb3a1cdb54b757405d90e15ad22f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal\" (UID: \"12bcfb3a1cdb54b757405d90e15ad22f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:58:36.124737 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.124707 7059 apiserver.go:52] "Watching apiserver" Nov 04 00:58:36.133249 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.133224 7059 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 04 00:58:36.135409 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.135371 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hhndm","openshift-multus/multus-additional-cni-plugins-jx882","openshift-monitoring/metrics-server-845fbdd675-9rvps","openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg","openshift-network-diagnostics/network-check-target-289kb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal","kube-system/global-pull-secret-syncer-2x7kl","openshift-console/console-69dd78b66f-clp8w","openshift-insights/insights-runtime-extractor-7jm5h","openshift-monitoring/monitoring-plugin-77f4f76594-4z274","openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25","openshift-monitoring/openshift-state-metrics-7bd7489957-dl2dc","openshift-network-operator/iptables-alerter-96mj9","openshift-monitoring/prometheus-k8s-0","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq","openshift-cluster-node-tuning-operator/tuned-q8w8s","openshift-monitoring/node-exporter-mhb4l","kube-system/kube-apiserver-proxy-ip-10-0-128-129.us-east-2.compute.internal","openshift-ingress-canary/ingress-canary-cv4w2","kube-system/konnectivity-agent-nb955","openshift-multus/network-metrics-daemon-l4d28","openshift-dns/node-resolver-h4s87","openshift-monitoring/alertmanager-main-0","openshift-multus/multus-7z64n","openshift-image-registry/node-ca-z4snb","openshift-ovn-kubernetes/ovnkube-node-xwbdv"] Nov 04 00:58:36.141811 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.141785 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:58:36.141811 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.141791 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 04 00:58:36.141991 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.141791 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 04 00:58:36.141991 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.141977 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 04 00:58:36.142259 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.142238 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 04 00:58:36.142357 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.142277 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 04 00:58:36.142477 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.142454 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 04 00:58:36.142535 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.142456 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 04 00:58:36.142535 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.142486 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 04 00:58:36.142631 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.142589 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 04 00:58:36.142631 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.142627 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 04 00:58:36.142722 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.142630 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 04 00:58:36.142722 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.142681 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 04 00:58:36.142821 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.142770 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 04 00:58:36.142902 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.142776 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 04 00:58:36.143089 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.143065 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 04 00:58:36.143189 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.143116 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 04 00:58:36.143242 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.143229 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 04 00:58:36.143305 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.143295 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 04 00:58:36.143618 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.143594 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 04 00:58:36.143940 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.143909 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-nszlw\"" Nov 04 00:58:36.144003 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.143910 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4rmxc\"" Nov 04 00:58:36.144147 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.144127 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sknxk\"" Nov 04 00:58:36.144236 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.144214 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zgrbh\"" Nov 04 00:58:36.144380 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.144361 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9jwd5\"" Nov 04 00:58:36.144380 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.144374 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 04 00:58:36.144820 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.144803 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-xdt54\"" Nov 04 00:58:36.145106 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.145086 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 00:58:36.145382 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.145359 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q48lg\"" Nov 04 00:58:36.145750 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.145731 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 04 00:58:36.145809 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.145769 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 04 00:58:36.146469 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.146440 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 04 00:58:36.146568 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.146469 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:58:36.147093 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.147071 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4k8lm\"" Nov 04 00:58:36.147330 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.147306 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-psr9r\"" Nov 04 00:58:36.147582 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.147558 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 00:58:36.147863 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.147559 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 04 00:58:36.148218 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.148201 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 04 00:58:36.148357 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.148339 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 04 00:58:36.148607 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.148575 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zx855\"" Nov 04 00:58:36.148668 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.148653 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 04 00:58:36.148786 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.148767 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 04 00:58:36.148878 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.148854 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 04 00:58:36.149068 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.148880 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 04 00:58:36.149189 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.149136 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-gtbz7\"" Nov 04 00:58:36.149372 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.149297 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 04 00:58:36.149372 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.149307 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 04 00:58:36.149372 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.149377 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 04 00:58:36.149585 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.149414 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 04 00:58:36.149585 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.149543 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 04 00:58:36.149725 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.149691 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 04 00:58:36.149869 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.149851 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kqwbk\"" Nov 04 00:58:36.150130 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.150036 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 04 00:58:36.150130 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.150057 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ldjm6\"" Nov 04 00:58:36.150130 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.150033 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 04 00:58:36.150130 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.148261 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 04 00:58:36.150345 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.150072 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 04 00:58:36.150407 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.150364 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xz2sx\"" Nov 04 00:58:36.150519 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.150501 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 04 00:58:36.150573 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.150530 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 04 00:58:36.150671 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.150651 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-pb69q\"" Nov 04 00:58:36.150671 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.150667 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 04 00:58:36.150879 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.150703 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 04 00:58:36.150972 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.150903 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 04 00:58:36.151340 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.151316 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 04 00:58:36.151419 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.151354 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 04 00:58:36.151419 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.151389 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 04 00:58:36.151546 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.151438 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 04 00:58:36.151546 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.151466 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 04 00:58:36.151546 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.151486 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 04 00:58:36.151546 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.151324 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 04 00:58:36.151767 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.151610 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 04 00:58:36.151767 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.151640 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 04 00:58:36.151857 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.151782 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-llw9w\"" Nov 04 00:58:36.151857 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.151842 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 04 00:58:36.151973 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.151866 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 04 00:58:36.151973 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.151892 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 04 00:58:36.151973 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.151952 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-f8bj3puo34rtq\"" Nov 04 00:58:36.152099 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.152008 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ln4qv\"" Nov 04 00:58:36.152099 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.152040 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 04 00:58:36.152099 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.152082 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 04 00:58:36.152246 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.152146 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5llll\"" Nov 04 00:58:36.153111 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.153083 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cjss9\"" Nov 04 00:58:36.153179 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.153160 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-rpsh2\"" Nov 04 00:58:36.153304 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.153289 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 04 00:58:36.153940 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.153908 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 04 00:58:36.156961 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.156905 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 04 00:58:36.157050 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.157036 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-qw5ml\"" Nov 04 00:58:36.157967 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.157952 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 04 00:58:36.158145 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.158129 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-97qhl\"" Nov 04 00:58:36.158205 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.158162 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 04 00:58:36.158726 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.158682 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-q5sdf\"" Nov 04 00:58:36.158726 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.158690 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 04 00:58:36.158866 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.158835 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-en40opnp7to2v\"" Nov 04 00:58:36.158866 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.158837 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 04 00:58:36.159089 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.158852 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 04 00:58:36.159089 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.158983 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 04 00:58:36.159089 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.159071 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 04 00:58:36.159254 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.159087 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 04 00:58:36.159254 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.159115 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 04 00:58:36.159454 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.159435 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 04 00:58:36.159760 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.159747 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 04 00:58:36.160388 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.160375 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wzbxc\"" Nov 04 00:58:36.160602 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.160584 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 04 00:58:36.160602 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.160594 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 04 00:58:36.160744 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.160633 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 04 00:58:36.160744 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.160673 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 04 00:58:36.161505 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.161485 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 04 00:58:36.161586 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.161547 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 04 00:58:36.161586 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.161582 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 04 00:58:36.161718 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.161706 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 04 00:58:36.161758 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.161725 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 04 00:58:36.161758 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.161726 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 04 00:58:36.162118 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.162086 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 04 00:58:36.162118 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.162087 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 04 00:58:36.162258 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.162246 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 04 00:58:36.162813 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.162800 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9outkgf7grlms\"" Nov 04 00:58:36.163517 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.163486 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/388103ca-b2f5-42b2-96ce-737f1ef00dc1-tmp-dir\") pod \"node-resolver-h4s87\" (UID: \"388103ca-b2f5-42b2-96ce-737f1ef00dc1\") " pod="openshift-dns/node-resolver-h4s87" Nov 04 00:58:36.163517 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.163510 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bac70f0f-4585-42fe-abba-916ac41fcc62-original-pull-secret\") pod \"global-pull-secret-syncer-2x7kl\" (UID: \"bac70f0f-4585-42fe-abba-916ac41fcc62\") " pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:58:36.163652 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.163528 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-host\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.163652 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.163543 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-run-k8s-cni-cncf-io\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.163652 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.163559 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-run-netns\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.163652 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.163577 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/26470ab5-58d3-4b17-bb82-e1b20803e7db-env-overrides\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.163652 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.163614 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-sysctl-conf\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.163877 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.163655 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-registration-dir\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:58:36.163877 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.163705 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d08dca6d-7563-4d2c-ae92-e7ffd4beda06-config-volume\") pod \"dns-default-hhndm\" (UID: \"d08dca6d-7563-4d2c-ae92-e7ffd4beda06\") " pod="openshift-dns/dns-default-hhndm" Nov 04 00:58:36.163877 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.163740 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/388103ca-b2f5-42b2-96ce-737f1ef00dc1-tmp-dir\") pod \"node-resolver-h4s87\" (UID: \"388103ca-b2f5-42b2-96ce-737f1ef00dc1\") " pod="openshift-dns/node-resolver-h4s87" Nov 04 00:58:36.163877 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.163747 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-systemd-units\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.163877 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.163750 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/26470ab5-58d3-4b17-bb82-e1b20803e7db-env-overrides\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.163877 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.163792 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-var-lib-openvswitch\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.163877 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.163832 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb9nj\" (UniqueName: \"kubernetes.io/projected/61e0c4ee-edf6-4938-96c0-c79812e00840-kube-api-access-wb9nj\") pod \"node-ca-z4snb\" (UID: \"61e0c4ee-edf6-4938-96c0-c79812e00840\") " pod="openshift-image-registry/node-ca-z4snb" Nov 04 00:58:36.163877 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.163860 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69mrb\" (UniqueName: \"kubernetes.io/projected/79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe-kube-api-access-69mrb\") pod \"iptables-alerter-96mj9\" (UID: \"79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe\") " pod="openshift-network-operator/iptables-alerter-96mj9" Nov 04 00:58:36.164447 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.163887 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bbfd4978-1740-4bf1-b18b-f2d7665fa421-cnibin\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:58:36.164447 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.163906 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d08dca6d-7563-4d2c-ae92-e7ffd4beda06-config-volume\") pod \"dns-default-hhndm\" (UID: \"d08dca6d-7563-4d2c-ae92-e7ffd4beda06\") " pod="openshift-dns/dns-default-hhndm" Nov 04 00:58:36.164447 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.163915 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x756c\" (UniqueName: \"kubernetes.io/projected/bbfd4978-1740-4bf1-b18b-f2d7665fa421-kube-api-access-x756c\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:58:36.164447 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.163939 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bac70f0f-4585-42fe-abba-916ac41fcc62-original-pull-secret\") pod \"global-pull-secret-syncer-2x7kl\" (UID: \"bac70f0f-4585-42fe-abba-916ac41fcc62\") " pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:58:36.164447 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.163969 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-run-ovn\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.164447 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.164073 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59e9f4f9-590f-43b0-999d-1fad31779d16-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7jm5h\" (UID: \"59e9f4f9-590f-43b0-999d-1fad31779d16\") " pod="openshift-insights/insights-runtime-extractor-7jm5h" Nov 04 00:58:36.164447 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.164105 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d90c5697-ccdd-4845-8972-dbbd952dcc87-cert\") pod \"ingress-canary-cv4w2\" (UID: \"d90c5697-ccdd-4845-8972-dbbd952dcc87\") " pod="openshift-ingress-canary/ingress-canary-cv4w2" Nov 04 00:58:36.164447 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.164140 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-multus-conf-dir\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.164447 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.164245 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbfd4978-1740-4bf1-b18b-f2d7665fa421-system-cni-dir\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:58:36.164447 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.164276 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59e9f4f9-590f-43b0-999d-1fad31779d16-crio-socket\") pod \"insights-runtime-extractor-7jm5h\" (UID: \"59e9f4f9-590f-43b0-999d-1fad31779d16\") " pod="openshift-insights/insights-runtime-extractor-7jm5h" Nov 04 00:58:36.164447 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.164328 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-var-lib-kubelet\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.164447 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.164340 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d90c5697-ccdd-4845-8972-dbbd952dcc87-cert\") pod \"ingress-canary-cv4w2\" (UID: \"d90c5697-ccdd-4845-8972-dbbd952dcc87\") " pod="openshift-ingress-canary/ingress-canary-cv4w2" Nov 04 00:58:36.164447 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.164373 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe-iptables-alerter-script\") pod \"iptables-alerter-96mj9\" (UID: \"79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe\") " pod="openshift-network-operator/iptables-alerter-96mj9" Nov 04 00:58:36.164447 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.164393 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59e9f4f9-590f-43b0-999d-1fad31779d16-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7jm5h\" (UID: \"59e9f4f9-590f-43b0-999d-1fad31779d16\") " pod="openshift-insights/insights-runtime-extractor-7jm5h" Nov 04 00:58:36.164447 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.164411 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-system-cni-dir\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.164447 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.164436 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-multus-cni-dir\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.164463 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-multus-socket-dir-parent\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.164503 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cdcb63f9-d455-4105-8911-33dc1d3dedad-multus-daemon-config\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.164519 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/26470ab5-58d3-4b17-bb82-e1b20803e7db-ovnkube-config\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.164535 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-systemd\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.164549 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-etc-selinux\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.164568 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-sys-fs\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.164575 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe-iptables-alerter-script\") pod \"iptables-alerter-96mj9\" (UID: \"79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe\") " pod="openshift-network-operator/iptables-alerter-96mj9" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.164626 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cdcb63f9-d455-4105-8911-33dc1d3dedad-multus-daemon-config\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.164637 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-sysconfig\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.164665 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8d094b5-dae6-4655-9425-c10f62f41f07-tmp\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.164694 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7kw6\" (UniqueName: \"kubernetes.io/projected/d8d094b5-dae6-4655-9425-c10f62f41f07-kube-api-access-b7kw6\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.164729 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d7sj\" (UniqueName: \"kubernetes.io/projected/9481376a-a63f-46e2-a605-fe4bb019c331-kube-api-access-6d7sj\") pod \"network-check-target-289kb\" (UID: \"9481376a-a63f-46e2-a605-fe4bb019c331\") " pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.164759 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-etc-openvswitch\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.164761 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8d094b5-dae6-4655-9425-c10f62f41f07-tmp\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.164742 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/26470ab5-58d3-4b17-bb82-e1b20803e7db-ovnkube-config\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.164773 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-log-socket\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.164827 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxb5v\" (UniqueName: \"kubernetes.io/projected/5d232021-de4b-4240-8877-e99fe3563101-kube-api-access-qxb5v\") pod \"network-metrics-daemon-l4d28\" (UID: \"5d232021-de4b-4240-8877-e99fe3563101\") " pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.164877 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cdcb63f9-d455-4105-8911-33dc1d3dedad-cni-binary-copy\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.164969 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.164975 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpl2j\" (UniqueName: \"kubernetes.io/projected/26470ab5-58d3-4b17-bb82-e1b20803e7db-kube-api-access-rpl2j\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.164999 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59e9f4f9-590f-43b0-999d-1fad31779d16-data-volume\") pod \"insights-runtime-extractor-7jm5h\" (UID: \"59e9f4f9-590f-43b0-999d-1fad31779d16\") " pod="openshift-insights/insights-runtime-extractor-7jm5h" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165031 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bbfd4978-1740-4bf1-b18b-f2d7665fa421-cni-binary-copy\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165056 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/388103ca-b2f5-42b2-96ce-737f1ef00dc1-hosts-file\") pod \"node-resolver-h4s87\" (UID: \"388103ca-b2f5-42b2-96ce-737f1ef00dc1\") " pod="openshift-dns/node-resolver-h4s87" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165102 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-kubelet\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165102 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cdcb63f9-d455-4105-8911-33dc1d3dedad-cni-binary-copy\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165115 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59e9f4f9-590f-43b0-999d-1fad31779d16-data-volume\") pod \"insights-runtime-extractor-7jm5h\" (UID: \"59e9f4f9-590f-43b0-999d-1fad31779d16\") " pod="openshift-insights/insights-runtime-extractor-7jm5h" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165165 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-slash\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165191 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d232021-de4b-4240-8877-e99fe3563101-metrics-certs\") pod \"network-metrics-daemon-l4d28\" (UID: \"5d232021-de4b-4240-8877-e99fe3563101\") " pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165212 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bac70f0f-4585-42fe-abba-916ac41fcc62-dbus\") pod \"global-pull-secret-syncer-2x7kl\" (UID: \"bac70f0f-4585-42fe-abba-916ac41fcc62\") " pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165250 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76469\" (UniqueName: \"kubernetes.io/projected/59e9f4f9-590f-43b0-999d-1fad31779d16-kube-api-access-76469\") pod \"insights-runtime-extractor-7jm5h\" (UID: \"59e9f4f9-590f-43b0-999d-1fad31779d16\") " pod="openshift-insights/insights-runtime-extractor-7jm5h" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165284 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-modprobe-d\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165311 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-tuned\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165336 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-run-systemd\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165358 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-cni-netd\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165335 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bbfd4978-1740-4bf1-b18b-f2d7665fa421-cni-binary-copy\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165408 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-tuned\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165430 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d232021-de4b-4240-8877-e99fe3563101-metrics-certs\") pod \"network-metrics-daemon-l4d28\" (UID: \"5d232021-de4b-4240-8877-e99fe3563101\") " pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165415 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/de6f6762-66c6-42c0-bf0d-7966483a040d-agent-certs\") pod \"konnectivity-agent-nb955\" (UID: \"de6f6762-66c6-42c0-bf0d-7966483a040d\") " pod="kube-system/konnectivity-agent-nb955" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165472 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59e9f4f9-590f-43b0-999d-1fad31779d16-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7jm5h\" (UID: \"59e9f4f9-590f-43b0-999d-1fad31779d16\") " pod="openshift-insights/insights-runtime-extractor-7jm5h" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165490 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-lib-modules\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.165478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165505 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d08dca6d-7563-4d2c-ae92-e7ffd4beda06-tmp-dir\") pod \"dns-default-hhndm\" (UID: \"d08dca6d-7563-4d2c-ae92-e7ffd4beda06\") " pod="openshift-dns/dns-default-hhndm" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165529 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld5mg\" (UniqueName: \"kubernetes.io/projected/cdcb63f9-d455-4105-8911-33dc1d3dedad-kube-api-access-ld5mg\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165534 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165630 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/26470ab5-58d3-4b17-bb82-e1b20803e7db-ovn-node-metrics-cert\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165648 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/de6f6762-66c6-42c0-bf0d-7966483a040d-agent-certs\") pod \"konnectivity-agent-nb955\" (UID: \"de6f6762-66c6-42c0-bf0d-7966483a040d\") " pod="kube-system/konnectivity-agent-nb955" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165656 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-node-log\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165662 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d08dca6d-7563-4d2c-ae92-e7ffd4beda06-tmp-dir\") pod \"dns-default-hhndm\" (UID: \"d08dca6d-7563-4d2c-ae92-e7ffd4beda06\") " pod="openshift-dns/dns-default-hhndm" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165694 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-run-ovn-kubernetes\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165722 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59e9f4f9-590f-43b0-999d-1fad31779d16-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7jm5h\" (UID: \"59e9f4f9-590f-43b0-999d-1fad31779d16\") " pod="openshift-insights/insights-runtime-extractor-7jm5h" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165733 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znlq8\" (UniqueName: \"kubernetes.io/projected/d08dca6d-7563-4d2c-ae92-e7ffd4beda06-kube-api-access-znlq8\") pod \"dns-default-hhndm\" (UID: \"d08dca6d-7563-4d2c-ae92-e7ffd4beda06\") " pod="openshift-dns/dns-default-hhndm" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165792 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zrwn\" (UniqueName: \"kubernetes.io/projected/d90c5697-ccdd-4845-8972-dbbd952dcc87-kube-api-access-2zrwn\") pod \"ingress-canary-cv4w2\" (UID: \"d90c5697-ccdd-4845-8972-dbbd952dcc87\") " pod="openshift-ingress-canary/ingress-canary-cv4w2" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165828 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-var-lib-cni-multus\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165854 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-hostroot\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165868 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/26470ab5-58d3-4b17-bb82-e1b20803e7db-ovn-node-metrics-cert\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165879 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-run-multus-certs\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165937 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-sysctl-d\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.165980 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-var-lib-cni-bin\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.166041 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-var-lib-kubelet\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.166069 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-etc-kubernetes\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.166129 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bbfd4978-1740-4bf1-b18b-f2d7665fa421-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.166163 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.166220 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-socket-dir\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.166253 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-device-dir\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.166274 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/26470ab5-58d3-4b17-bb82-e1b20803e7db-ovnkube-script-lib\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.166291 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bbfd4978-1740-4bf1-b18b-f2d7665fa421-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.166312 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/61e0c4ee-edf6-4938-96c0-c79812e00840-serviceca\") pod \"node-ca-z4snb\" (UID: \"61e0c4ee-edf6-4938-96c0-c79812e00840\") " pod="openshift-image-registry/node-ca-z4snb" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.166340 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-run\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.166365 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bbfd4978-1740-4bf1-b18b-f2d7665fa421-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.166394 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bbfd4978-1740-4bf1-b18b-f2d7665fa421-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.166419 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/26470ab5-58d3-4b17-bb82-e1b20803e7db-ovnkube-script-lib\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.166420 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmn8n\" (UniqueName: \"kubernetes.io/projected/d2344442-689b-49f6-a97b-365500025b2d-kube-api-access-wmn8n\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.166524 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-run-openvswitch\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.166570 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.166583 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bbfd4978-1740-4bf1-b18b-f2d7665fa421-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.166611 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/de6f6762-66c6-42c0-bf0d-7966483a040d-konnectivity-ca\") pod \"konnectivity-agent-nb955\" (UID: \"de6f6762-66c6-42c0-bf0d-7966483a040d\") " pod="kube-system/konnectivity-agent-nb955" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.166635 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61e0c4ee-edf6-4938-96c0-c79812e00840-host\") pod \"node-ca-z4snb\" (UID: \"61e0c4ee-edf6-4938-96c0-c79812e00840\") " pod="openshift-image-registry/node-ca-z4snb" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.166658 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-kubernetes\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.166681 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-sys\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.166711 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-cni-bin\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.166777 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfzkw\" (UniqueName: \"kubernetes.io/projected/388103ca-b2f5-42b2-96ce-737f1ef00dc1-kube-api-access-sfzkw\") pod \"node-resolver-h4s87\" (UID: \"388103ca-b2f5-42b2-96ce-737f1ef00dc1\") " pod="openshift-dns/node-resolver-h4s87" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.166809 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-run-netns\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.166872 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bac70f0f-4585-42fe-abba-916ac41fcc62-kubelet-config\") pod \"global-pull-secret-syncer-2x7kl\" (UID: \"bac70f0f-4585-42fe-abba-916ac41fcc62\") " pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.166887 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/61e0c4ee-edf6-4938-96c0-c79812e00840-serviceca\") pod \"node-ca-z4snb\" (UID: \"61e0c4ee-edf6-4938-96c0-c79812e00840\") " pod="openshift-image-registry/node-ca-z4snb" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.166879 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/de6f6762-66c6-42c0-bf0d-7966483a040d-konnectivity-ca\") pod \"konnectivity-agent-nb955\" (UID: \"de6f6762-66c6-42c0-bf0d-7966483a040d\") " pod="kube-system/konnectivity-agent-nb955" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.166948 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d08dca6d-7563-4d2c-ae92-e7ffd4beda06-metrics-tls\") pod \"dns-default-hhndm\" (UID: \"d08dca6d-7563-4d2c-ae92-e7ffd4beda06\") " pod="openshift-dns/dns-default-hhndm" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.166975 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe-host-slash\") pod \"iptables-alerter-96mj9\" (UID: \"79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe\") " pod="openshift-network-operator/iptables-alerter-96mj9" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.167021 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-cnibin\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.167237 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.167143 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d08dca6d-7563-4d2c-ae92-e7ffd4beda06-metrics-tls\") pod \"dns-default-hhndm\" (UID: \"d08dca6d-7563-4d2c-ae92-e7ffd4beda06\") " pod="openshift-dns/dns-default-hhndm" Nov 04 00:58:36.170161 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.170144 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69mrb\" (UniqueName: \"kubernetes.io/projected/79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe-kube-api-access-69mrb\") pod \"iptables-alerter-96mj9\" (UID: \"79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe\") " pod="openshift-network-operator/iptables-alerter-96mj9" Nov 04 00:58:36.170613 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.170578 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x756c\" (UniqueName: \"kubernetes.io/projected/bbfd4978-1740-4bf1-b18b-f2d7665fa421-kube-api-access-x756c\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:58:36.170678 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.170663 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb9nj\" (UniqueName: \"kubernetes.io/projected/61e0c4ee-edf6-4938-96c0-c79812e00840-kube-api-access-wb9nj\") pod \"node-ca-z4snb\" (UID: \"61e0c4ee-edf6-4938-96c0-c79812e00840\") " pod="openshift-image-registry/node-ca-z4snb" Nov 04 00:58:36.171373 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.171356 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7kw6\" (UniqueName: \"kubernetes.io/projected/d8d094b5-dae6-4655-9425-c10f62f41f07-kube-api-access-b7kw6\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.171481 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.171462 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpl2j\" (UniqueName: \"kubernetes.io/projected/26470ab5-58d3-4b17-bb82-e1b20803e7db-kube-api-access-rpl2j\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.171741 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.171687 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76469\" (UniqueName: \"kubernetes.io/projected/59e9f4f9-590f-43b0-999d-1fad31779d16-kube-api-access-76469\") pod \"insights-runtime-extractor-7jm5h\" (UID: \"59e9f4f9-590f-43b0-999d-1fad31779d16\") " pod="openshift-insights/insights-runtime-extractor-7jm5h" Nov 04 00:58:36.171849 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.171828 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d7sj\" (UniqueName: \"kubernetes.io/projected/9481376a-a63f-46e2-a605-fe4bb019c331-kube-api-access-6d7sj\") pod \"network-check-target-289kb\" (UID: \"9481376a-a63f-46e2-a605-fe4bb019c331\") " pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:58:36.172037 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.172018 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxb5v\" (UniqueName: \"kubernetes.io/projected/5d232021-de4b-4240-8877-e99fe3563101-kube-api-access-qxb5v\") pod \"network-metrics-daemon-l4d28\" (UID: \"5d232021-de4b-4240-8877-e99fe3563101\") " pod="openshift-multus/network-metrics-daemon-l4d28" Nov 04 00:58:36.172147 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.172123 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld5mg\" (UniqueName: \"kubernetes.io/projected/cdcb63f9-d455-4105-8911-33dc1d3dedad-kube-api-access-ld5mg\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.172232 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.172211 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfzkw\" (UniqueName: \"kubernetes.io/projected/388103ca-b2f5-42b2-96ce-737f1ef00dc1-kube-api-access-sfzkw\") pod \"node-resolver-h4s87\" (UID: \"388103ca-b2f5-42b2-96ce-737f1ef00dc1\") " pod="openshift-dns/node-resolver-h4s87" Nov 04 00:58:36.172285 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.172272 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmn8n\" (UniqueName: \"kubernetes.io/projected/d2344442-689b-49f6-a97b-365500025b2d-kube-api-access-wmn8n\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:58:36.172394 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.172377 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znlq8\" (UniqueName: \"kubernetes.io/projected/d08dca6d-7563-4d2c-ae92-e7ffd4beda06-kube-api-access-znlq8\") pod \"dns-default-hhndm\" (UID: \"d08dca6d-7563-4d2c-ae92-e7ffd4beda06\") " pod="openshift-dns/dns-default-hhndm" Nov 04 00:58:36.172697 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.172682 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zrwn\" (UniqueName: \"kubernetes.io/projected/d90c5697-ccdd-4845-8972-dbbd952dcc87-kube-api-access-2zrwn\") pod \"ingress-canary-cv4w2\" (UID: \"d90c5697-ccdd-4845-8972-dbbd952dcc87\") " pod="openshift-ingress-canary/ingress-canary-cv4w2" Nov 04 00:58:36.241726 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.241700 7059 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:58:36.241869 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.241752 7059 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:58:36.243493 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.243470 7059 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 04 00:58:36.253349 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.253329 7059 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 00:58:36.253457 ip-10-0-128-129 kubenswrapper[7059]: E1104 00:58:36.253373 7059 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-129.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:58:36.253700 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.253684 7059 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 00:58:36.253767 ip-10-0-128-129 kubenswrapper[7059]: E1104 00:58:36.253722 7059 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-129.us-east-2.compute.internal" Nov 04 00:58:36.268078 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268053 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59m4r\" (UniqueName: \"kubernetes.io/projected/5522ff12-fed1-4411-96b0-1cceb6f0b634-kube-api-access-59m4r\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:36.268138 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268080 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-etc-openvswitch\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.268138 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268097 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-log-socket\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.268138 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268111 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.268244 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268152 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-log-socket\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.268244 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268175 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:36.268244 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268172 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-etc-openvswitch\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.268244 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268189 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-web-config\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.268244 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268227 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.268517 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268257 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/388103ca-b2f5-42b2-96ce-737f1ef00dc1-hosts-file\") pod \"node-resolver-h4s87\" (UID: \"388103ca-b2f5-42b2-96ce-737f1ef00dc1\") " pod="openshift-dns/node-resolver-h4s87" Nov 04 00:58:36.268517 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268282 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-slash\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.268517 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268305 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:58:36.268517 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268372 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:58:36.268517 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268389 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-web-config\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.268517 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268387 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/388103ca-b2f5-42b2-96ce-737f1ef00dc1-hosts-file\") pod \"node-resolver-h4s87\" (UID: \"388103ca-b2f5-42b2-96ce-737f1ef00dc1\") " pod="openshift-dns/node-resolver-h4s87" Nov 04 00:58:36.268517 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268359 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-slash\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.268517 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268408 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-device-dir\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:58:36.268517 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268454 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95jzn\" (UniqueName: \"kubernetes.io/projected/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-kube-api-access-95jzn\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.268517 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268478 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-run-systemd\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.268517 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268492 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-device-dir\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:58:36.268517 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268504 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-lib-modules\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.268517 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268515 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.269109 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268528 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bbfd4978-1740-4bf1-b18b-f2d7665fa421-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:58:36.269109 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268534 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-run-systemd\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.269109 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268561 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:36.269109 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268586 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-node-log\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.269109 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268593 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bbfd4978-1740-4bf1-b18b-f2d7665fa421-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:58:36.269109 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268605 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:36.269109 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268610 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-run-ovn-kubernetes\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.269109 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268654 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-lib-modules\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.269109 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268640 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-node-log\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.269109 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268678 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-var-lib-cni-multus\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.269109 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268714 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-config-out\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:36.269109 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268738 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-run-ovn-kubernetes\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.269109 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268740 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-sysctl-d\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.269109 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268754 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-var-lib-cni-multus\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.269109 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268792 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-config-out\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:36.269109 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268787 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-var-lib-cni-bin\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.269109 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268833 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-socket-dir\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:58:36.269109 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268833 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:36.269109 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268863 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-sysctl-d\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.269109 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268871 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-dl2dc\" (UID: \"1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dl2dc" Nov 04 00:58:36.269109 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268876 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-var-lib-cni-bin\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.269109 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268887 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-socket-dir\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:58:36.269109 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268918 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/559e1de2-7e21-4aeb-916f-7a75e0c75841-audit-log\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:36.269109 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268961 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:36.269109 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.268978 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-dl2dc\" (UID: \"1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dl2dc" Nov 04 00:58:36.269109 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269006 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/559e1de2-7e21-4aeb-916f-7a75e0c75841-secret-metrics-server-tls\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:36.269109 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269033 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/559e1de2-7e21-4aeb-916f-7a75e0c75841-metrics-server-audit-profiles\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:36.269109 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269042 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/559e1de2-7e21-4aeb-916f-7a75e0c75841-audit-log\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:36.269109 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269060 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:36.269109 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269097 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-dl2dc\" (UID: \"1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dl2dc" Nov 04 00:58:36.269109 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269100 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sprpf\" (UniqueName: \"kubernetes.io/projected/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-kube-api-access-sprpf\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:36.269109 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269128 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269149 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-oauth-serving-cert\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269176 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-run-openvswitch\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269189 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-dl2dc\" (UID: \"1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dl2dc" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269204 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/95638663-fffb-41a6-8056-c1f4293a3fb0-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-8wz25\" (UID: \"95638663-fffb-41a6-8056-c1f4293a3fb0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269230 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269235 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-run-openvswitch\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269256 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269283 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-cni-bin\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269178 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269325 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-oauth-serving-cert\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269307 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bac70f0f-4585-42fe-abba-916ac41fcc62-kubelet-config\") pod \"global-pull-secret-syncer-2x7kl\" (UID: \"bac70f0f-4585-42fe-abba-916ac41fcc62\") " pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269362 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-cni-bin\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269376 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe-host-slash\") pod \"iptables-alerter-96mj9\" (UID: \"79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe\") " pod="openshift-network-operator/iptables-alerter-96mj9" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269409 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5522ff12-fed1-4411-96b0-1cceb6f0b634-node-exporter-wtmp\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269443 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe-host-slash\") pod \"iptables-alerter-96mj9\" (UID: \"79b6b73a-fb4d-4a89-8d7e-efbb3c463bfe\") " pod="openshift-network-operator/iptables-alerter-96mj9" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269410 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/559e1de2-7e21-4aeb-916f-7a75e0c75841-metrics-server-audit-profiles\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269411 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bac70f0f-4585-42fe-abba-916ac41fcc62-kubelet-config\") pod \"global-pull-secret-syncer-2x7kl\" (UID: \"bac70f0f-4585-42fe-abba-916ac41fcc62\") " pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269439 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/95638663-fffb-41a6-8056-c1f4293a3fb0-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-8wz25\" (UID: \"95638663-fffb-41a6-8056-c1f4293a3fb0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269443 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-host\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269486 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-host\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269501 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269507 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-run-k8s-cni-cncf-io\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269532 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-run-k8s-cni-cncf-io\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269542 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269555 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5522ff12-fed1-4411-96b0-1cceb6f0b634-metrics-client-ca\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269579 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/559e1de2-7e21-4aeb-916f-7a75e0c75841-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269596 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/559e1de2-7e21-4aeb-916f-7a75e0c75841-secret-metrics-server-tls\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269602 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269649 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5ssp\" (UniqueName: \"kubernetes.io/projected/5bcf3164-e6e8-4323-aad0-ee8adbe4a77d-kube-api-access-n5ssp\") pod \"monitoring-plugin-77f4f76594-4z274\" (UID: \"5bcf3164-e6e8-4323-aad0-ee8adbe4a77d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4z274" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269679 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-sysctl-conf\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269702 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-registration-dir\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269727 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5522ff12-fed1-4411-96b0-1cceb6f0b634-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269733 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/559e1de2-7e21-4aeb-916f-7a75e0c75841-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269739 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269751 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/559e1de2-7e21-4aeb-916f-7a75e0c75841-client-ca-bundle\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269762 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5522ff12-fed1-4411-96b0-1cceb6f0b634-metrics-client-ca\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269778 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/95638663-fffb-41a6-8056-c1f4293a3fb0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-8wz25\" (UID: \"95638663-fffb-41a6-8056-c1f4293a3fb0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269787 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-registration-dir\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269822 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-sysctl-conf\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269967 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/559e1de2-7e21-4aeb-916f-7a75e0c75841-client-ca-bundle\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269970 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.269997 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5522ff12-fed1-4411-96b0-1cceb6f0b634-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270011 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270037 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/95638663-fffb-41a6-8056-c1f4293a3fb0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-8wz25\" (UID: \"95638663-fffb-41a6-8056-c1f4293a3fb0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270040 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-dl2dc\" (UID: \"1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dl2dc" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270068 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-config-out\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270093 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270121 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270149 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270161 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-config-out\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270152 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfpdd\" (UniqueName: \"kubernetes.io/projected/559e1de2-7e21-4aeb-916f-7a75e0c75841-kube-api-access-bfpdd\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270213 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270240 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-config\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270249 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270266 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-multus-conf-dir\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270295 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95638663-fffb-41a6-8056-c1f4293a3fb0-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-8wz25\" (UID: \"95638663-fffb-41a6-8056-c1f4293a3fb0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270320 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270237 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-dl2dc\" (UID: \"1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dl2dc" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270331 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270346 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/34aecbea-ff97-4f23-8336-326316720a2f-console-serving-cert\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270372 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270384 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-multus-conf-dir\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270397 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prd7v\" (UniqueName: \"kubernetes.io/projected/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-kube-api-access-prd7v\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270426 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-var-lib-kubelet\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270452 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-multus-socket-dir-parent\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270477 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-systemd\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270502 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-etc-selinux\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270526 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270526 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-sys-fs\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270582 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270589 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-sys-fs\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270338 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270616 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270621 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-sysconfig\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270666 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-multus-socket-dir-parent\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270679 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d2344442-689b-49f6-a97b-365500025b2d-etc-selinux\") pod \"aws-ebs-csi-driver-node-wt4cq\" (UID: \"d2344442-689b-49f6-a97b-365500025b2d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wt4cq" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270671 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-sysconfig\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270671 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-systemd\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270582 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95638663-fffb-41a6-8056-c1f4293a3fb0-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-8wz25\" (UID: \"95638663-fffb-41a6-8056-c1f4293a3fb0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270708 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270779 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-config\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270790 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-service-ca\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:36.270765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270826 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270846 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270858 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-kubelet\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270883 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bac70f0f-4585-42fe-abba-916ac41fcc62-dbus\") pod \"global-pull-secret-syncer-2x7kl\" (UID: \"bac70f0f-4585-42fe-abba-916ac41fcc62\") " pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270908 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-modprobe-d\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270955 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/95638663-fffb-41a6-8056-c1f4293a3fb0-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-8wz25\" (UID: \"95638663-fffb-41a6-8056-c1f4293a3fb0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270990 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-service-ca\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270985 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-web-config\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271013 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bac70f0f-4585-42fe-abba-916ac41fcc62-dbus\") pod \"global-pull-secret-syncer-2x7kl\" (UID: \"bac70f0f-4585-42fe-abba-916ac41fcc62\") " pod="kube-system/global-pull-secret-syncer-2x7kl" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271034 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-cni-netd\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271044 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-kubelet\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271044 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271063 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-console-config\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.270952 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271084 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/95638663-fffb-41a6-8056-c1f4293a3fb0-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-8wz25\" (UID: \"95638663-fffb-41a6-8056-c1f4293a3fb0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271101 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271128 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271113 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-cni-netd\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271148 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-var-lib-kubelet\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271159 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-modprobe-d\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271156 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-hostroot\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271170 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271202 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-run-multus-certs\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271242 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5bcf3164-e6e8-4323-aad0-ee8adbe4a77d-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4z274\" (UID: \"5bcf3164-e6e8-4323-aad0-ee8adbe4a77d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4z274" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271266 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34aecbea-ff97-4f23-8336-326316720a2f-tmp\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271269 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-console-config\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271286 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-run-multus-certs\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271290 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271299 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271329 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-var-lib-kubelet\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271340 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34aecbea-ff97-4f23-8336-326316720a2f-tmp\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271358 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-etc-kubernetes\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271385 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-var-lib-kubelet\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271387 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr5ms\" (UniqueName: \"kubernetes.io/projected/1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84-kube-api-access-wr5ms\") pod \"openshift-state-metrics-7bd7489957-dl2dc\" (UID: \"1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dl2dc" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271422 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5bcf3164-e6e8-4323-aad0-ee8adbe4a77d-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4z274\" (UID: \"5bcf3164-e6e8-4323-aad0-ee8adbe4a77d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4z274" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271421 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5522ff12-fed1-4411-96b0-1cceb6f0b634-node-exporter-accelerators-collector-config\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271446 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-etc-kubernetes\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271425 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271535 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-hostroot\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271599 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271611 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5522ff12-fed1-4411-96b0-1cceb6f0b634-node-exporter-accelerators-collector-config\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271619 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-web-config\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271628 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/34aecbea-ff97-4f23-8336-326316720a2f-console-serving-cert\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271625 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271631 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271694 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-run\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271722 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqzsj\" (UniqueName: \"kubernetes.io/projected/95638663-fffb-41a6-8056-c1f4293a3fb0-kube-api-access-hqzsj\") pod \"kube-state-metrics-5dc8d7d89b-8wz25\" (UID: \"95638663-fffb-41a6-8056-c1f4293a3fb0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271748 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-thanos-querier-tls\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271761 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-run\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271777 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271804 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61e0c4ee-edf6-4938-96c0-c79812e00840-host\") pod \"node-ca-z4snb\" (UID: \"61e0c4ee-edf6-4938-96c0-c79812e00840\") " pod="openshift-image-registry/node-ca-z4snb" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271812 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271777 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271827 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-kubernetes\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271823 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271850 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-sys\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271864 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61e0c4ee-edf6-4938-96c0-c79812e00840-host\") pod \"node-ca-z4snb\" (UID: \"61e0c4ee-edf6-4938-96c0-c79812e00840\") " pod="openshift-image-registry/node-ca-z4snb" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271874 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/559e1de2-7e21-4aeb-916f-7a75e0c75841-secret-metrics-server-client-certs\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271890 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-etc-kubernetes\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271900 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-run-netns\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271913 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8d094b5-dae6-4655-9425-c10f62f41f07-sys\") pod \"tuned-q8w8s\" (UID: \"d8d094b5-dae6-4655-9425-c10f62f41f07\") " pod="openshift-cluster-node-tuning-operator/tuned-q8w8s" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271954 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-cnibin\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271957 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-thanos-querier-tls\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.271977 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-host-run-netns\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272000 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-cnibin\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272007 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5522ff12-fed1-4411-96b0-1cceb6f0b634-node-exporter-textfile\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272044 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272064 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-run-netns\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272065 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5522ff12-fed1-4411-96b0-1cceb6f0b634-node-exporter-textfile\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272081 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5522ff12-fed1-4411-96b0-1cceb6f0b634-root\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272097 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/559e1de2-7e21-4aeb-916f-7a75e0c75841-secret-metrics-server-client-certs\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272108 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-host-run-netns\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272112 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/34aecbea-ff97-4f23-8336-326316720a2f-console-oauth-config\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272145 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-metrics-client-ca\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272223 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-tmp\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272274 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5522ff12-fed1-4411-96b0-1cceb6f0b634-sys\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272303 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272309 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272306 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/34aecbea-ff97-4f23-8336-326316720a2f-console-oauth-config\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272335 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-tmp\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272355 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272451 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-metrics-client-ca\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272460 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-systemd-units\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272486 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272493 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272522 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-var-lib-openvswitch\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272491 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-systemd-units\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272494 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-var-lib-openvswitch\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272558 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bbfd4978-1740-4bf1-b18b-f2d7665fa421-cnibin\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272574 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-config-volume\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272588 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-run-ovn\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272607 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbfd4978-1740-4bf1-b18b-f2d7665fa421-system-cni-dir\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272630 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bbfd4978-1740-4bf1-b18b-f2d7665fa421-cnibin\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272631 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5522ff12-fed1-4411-96b0-1cceb6f0b634-node-exporter-tls\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272659 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbfd4978-1740-4bf1-b18b-f2d7665fa421-system-cni-dir\") pod \"multus-additional-cni-plugins-jx882\" (UID: \"bbfd4978-1740-4bf1-b18b-f2d7665fa421\") " pod="openshift-multus/multus-additional-cni-plugins-jx882" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272637 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/26470ab5-58d3-4b17-bb82-e1b20803e7db-run-ovn\") pod \"ovnkube-node-xwbdv\" (UID: \"26470ab5-58d3-4b17-bb82-e1b20803e7db\") " pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272676 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95638663-fffb-41a6-8056-c1f4293a3fb0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-8wz25\" (UID: \"95638663-fffb-41a6-8056-c1f4293a3fb0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272721 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272742 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272748 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272789 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-config-volume\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272819 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59e9f4f9-590f-43b0-999d-1fad31779d16-crio-socket\") pod \"insights-runtime-extractor-7jm5h\" (UID: \"59e9f4f9-590f-43b0-999d-1fad31779d16\") " pod="openshift-insights/insights-runtime-extractor-7jm5h" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272846 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-system-cni-dir\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272861 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59e9f4f9-590f-43b0-999d-1fad31779d16-crio-socket\") pod \"insights-runtime-extractor-7jm5h\" (UID: \"59e9f4f9-590f-43b0-999d-1fad31779d16\") " pod="openshift-insights/insights-runtime-extractor-7jm5h" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272824 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5522ff12-fed1-4411-96b0-1cceb6f0b634-node-exporter-tls\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272870 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-multus-cni-dir\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272912 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79nvt\" (UniqueName: \"kubernetes.io/projected/34aecbea-ff97-4f23-8336-326316720a2f-kube-api-access-79nvt\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272940 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-multus-cni-dir\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272941 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272916 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cdcb63f9-d455-4105-8911-33dc1d3dedad-system-cni-dir\") pod \"multus-7z64n\" (UID: \"cdcb63f9-d455-4105-8911-33dc1d3dedad\") " pod="openshift-multus/multus-7z64n" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272967 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-trusted-ca-bundle\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272970 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.272909 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95638663-fffb-41a6-8056-c1f4293a3fb0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-8wz25\" (UID: \"95638663-fffb-41a6-8056-c1f4293a3fb0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.273114 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-grpc-tls\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.273277 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-secret-grpc-tls\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:36.274066 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.273300 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-trusted-ca-bundle\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:36.278160 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.277587 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59m4r\" (UniqueName: \"kubernetes.io/projected/5522ff12-fed1-4411-96b0-1cceb6f0b634-kube-api-access-59m4r\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:36.278160 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.278007 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95jzn\" (UniqueName: \"kubernetes.io/projected/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-kube-api-access-95jzn\") pod \"prometheus-k8s-0\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:36.279034 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.279014 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5ssp\" (UniqueName: \"kubernetes.io/projected/5bcf3164-e6e8-4323-aad0-ee8adbe4a77d-kube-api-access-n5ssp\") pod \"monitoring-plugin-77f4f76594-4z274\" (UID: \"5bcf3164-e6e8-4323-aad0-ee8adbe4a77d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4z274" Nov 04 00:58:36.279656 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.279636 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sprpf\" (UniqueName: \"kubernetes.io/projected/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-kube-api-access-sprpf\") pod \"alertmanager-main-0\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:36.279696 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.279638 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prd7v\" (UniqueName: \"kubernetes.io/projected/6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a-kube-api-access-prd7v\") pod \"thanos-querier-6d5d5b78fd-wjqgg\" (UID: \"6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a\") " pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:36.279696 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.279666 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfpdd\" (UniqueName: \"kubernetes.io/projected/559e1de2-7e21-4aeb-916f-7a75e0c75841-kube-api-access-bfpdd\") pod \"metrics-server-845fbdd675-9rvps\" (UID: \"559e1de2-7e21-4aeb-916f-7a75e0c75841\") " pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:36.280036 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.280020 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqzsj\" (UniqueName: \"kubernetes.io/projected/95638663-fffb-41a6-8056-c1f4293a3fb0-kube-api-access-hqzsj\") pod \"kube-state-metrics-5dc8d7d89b-8wz25\" (UID: \"95638663-fffb-41a6-8056-c1f4293a3fb0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8wz25" Nov 04 00:58:36.280322 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.280306 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr5ms\" (UniqueName: \"kubernetes.io/projected/1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84-kube-api-access-wr5ms\") pod \"openshift-state-metrics-7bd7489957-dl2dc\" (UID: \"1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dl2dc" Nov 04 00:58:36.281866 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.281851 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79nvt\" (UniqueName: \"kubernetes.io/projected/34aecbea-ff97-4f23-8336-326316720a2f-kube-api-access-79nvt\") pod \"console-69dd78b66f-clp8w\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:36.374259 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.374226 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5522ff12-fed1-4411-96b0-1cceb6f0b634-node-exporter-wtmp\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:36.374402 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.374290 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5522ff12-fed1-4411-96b0-1cceb6f0b634-root\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:36.374402 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.374306 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5522ff12-fed1-4411-96b0-1cceb6f0b634-sys\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:36.374402 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.374357 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5522ff12-fed1-4411-96b0-1cceb6f0b634-sys\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:36.374402 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.374393 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5522ff12-fed1-4411-96b0-1cceb6f0b634-node-exporter-wtmp\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:36.374569 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.374420 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5522ff12-fed1-4411-96b0-1cceb6f0b634-root\") pod \"node-exporter-mhb4l\" (UID: \"5522ff12-fed1-4411-96b0-1cceb6f0b634\") " pod="openshift-monitoring/node-exporter-mhb4l" Nov 04 00:58:36.479151 ip-10-0-128-129 kubenswrapper[7059]: E1104 00:58:36.479121 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 00:58:36.700095 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.700067 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:36.778288 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:36.778234 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hhndm" Nov 04 00:58:37.428484 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:37.428456 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:37.811124 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:37.811057 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:58:39.271568 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:39.271541 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6d5d5b78fd-wjqgg" Nov 04 00:58:39.486324 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:39.486302 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:58:39.486517 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:39.486505 7059 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 00:58:39.650241 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:39.650170 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4z274" Nov 04 00:58:40.147198 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:40.147166 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:58:45.156647 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:45.156574 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-845fbdd675-9rvps" Nov 04 00:58:49.644118 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:49.644050 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 04 00:58:49.644674 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:49.644449 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerName="alertmanager" containerID="cri-o://b461d5ec652164a2611aa3a9967068acd7e76e9cbca9e03152786644e970d82b" gracePeriod=120 Nov 04 00:58:49.644674 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:49.644497 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerName="kube-rbac-proxy-metric" containerID="cri-o://e5c8d787a70a6a5bb39ecdfa4b7346a21df9ed1b5e7546f30e0fef7669aaac9e" gracePeriod=120 Nov 04 00:58:49.644674 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:49.644548 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerName="config-reloader" containerID="cri-o://78cfe9bd056d53b1760fe77c2b25381fda1cf7e8e227c08c676986daffd8c2af" gracePeriod=120 Nov 04 00:58:49.644674 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:49.644548 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerName="kube-rbac-proxy" containerID="cri-o://adf1116db30959de5b988e09caba6de90c86c3d8eac0b536bc15e7d031ab3035" gracePeriod=120 Nov 04 00:58:49.644674 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:49.644505 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerName="prom-label-proxy" containerID="cri-o://6effbe28eed4891cb772305777544e80ea635a5a35153f8cfa55897562a48642" gracePeriod=120 Nov 04 00:58:49.644968 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:49.644678 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerName="kube-rbac-proxy-web" containerID="cri-o://444d87bd8b809346877521d4160630c88f94461cd7f5ceff86a439d468197199" gracePeriod=120 Nov 04 00:58:50.300187 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:50.300156 7059 generic.go:356] "Generic (PLEG): container finished" podID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerID="6effbe28eed4891cb772305777544e80ea635a5a35153f8cfa55897562a48642" exitCode=0 Nov 04 00:58:50.300187 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:50.300181 7059 generic.go:356] "Generic (PLEG): container finished" podID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerID="e5c8d787a70a6a5bb39ecdfa4b7346a21df9ed1b5e7546f30e0fef7669aaac9e" exitCode=0 Nov 04 00:58:50.300187 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:50.300188 7059 generic.go:356] "Generic (PLEG): container finished" podID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerID="adf1116db30959de5b988e09caba6de90c86c3d8eac0b536bc15e7d031ab3035" exitCode=0 Nov 04 00:58:50.300187 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:50.300194 7059 generic.go:356] "Generic (PLEG): container finished" podID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerID="78cfe9bd056d53b1760fe77c2b25381fda1cf7e8e227c08c676986daffd8c2af" exitCode=0 Nov 04 00:58:50.300441 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:50.300201 7059 generic.go:356] "Generic (PLEG): container finished" podID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerID="b461d5ec652164a2611aa3a9967068acd7e76e9cbca9e03152786644e970d82b" exitCode=0 Nov 04 00:58:50.300441 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:50.300226 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f45dd13e-3bfc-49ee-80d8-8252ec2f352c","Type":"ContainerDied","Data":"6effbe28eed4891cb772305777544e80ea635a5a35153f8cfa55897562a48642"} Nov 04 00:58:50.300441 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:50.300291 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f45dd13e-3bfc-49ee-80d8-8252ec2f352c","Type":"ContainerDied","Data":"e5c8d787a70a6a5bb39ecdfa4b7346a21df9ed1b5e7546f30e0fef7669aaac9e"} Nov 04 00:58:50.300441 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:50.300317 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f45dd13e-3bfc-49ee-80d8-8252ec2f352c","Type":"ContainerDied","Data":"adf1116db30959de5b988e09caba6de90c86c3d8eac0b536bc15e7d031ab3035"} Nov 04 00:58:50.300441 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:50.300341 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f45dd13e-3bfc-49ee-80d8-8252ec2f352c","Type":"ContainerDied","Data":"78cfe9bd056d53b1760fe77c2b25381fda1cf7e8e227c08c676986daffd8c2af"} Nov 04 00:58:50.300441 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:50.300363 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f45dd13e-3bfc-49ee-80d8-8252ec2f352c","Type":"ContainerDied","Data":"b461d5ec652164a2611aa3a9967068acd7e76e9cbca9e03152786644e970d82b"} Nov 04 00:58:50.905624 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:50.905602 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:50.995250 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:50.995185 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-config-volume\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-config-volume\") pod \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " Nov 04 00:58:50.995250 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:50.995225 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-main-tls\") pod \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " Nov 04 00:58:50.995442 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:50.995260 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-kube-rbac-proxy-web\") pod \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " Nov 04 00:58:50.995442 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:50.995282 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-kube-api-access-sprpf\" (UniqueName: \"kubernetes.io/projected/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-kube-api-access-sprpf\") pod \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " Nov 04 00:58:50.995442 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:50.995327 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " Nov 04 00:58:50.995442 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:50.995356 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-config-out\" (UniqueName: \"kubernetes.io/empty-dir/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-config-out\") pod \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " Nov 04 00:58:50.995442 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:50.995385 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-alertmanager-trusted-ca-bundle\") pod \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " Nov 04 00:58:50.995442 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:50.995415 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-tls-assets\" (UniqueName: \"kubernetes.io/projected/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-tls-assets\") pod \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " Nov 04 00:58:50.995442 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:50.995443 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-alertmanager-main-db\") pod \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " Nov 04 00:58:50.995805 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:50.995468 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-web-config\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-web-config\") pod \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " Nov 04 00:58:50.995805 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:50.995496 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-kube-rbac-proxy\") pod \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " Nov 04 00:58:50.995805 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:50.995567 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-metrics-client-ca\") pod \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " Nov 04 00:58:50.995805 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:50.995611 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-cluster-tls-config\") pod \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\" (UID: \"f45dd13e-3bfc-49ee-80d8-8252ec2f352c\") " Nov 04 00:58:50.997052 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:50.996418 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-alertmanager-trusted-ca-bundle" pod "f45dd13e-3bfc-49ee-80d8-8252ec2f352c" (UID: "f45dd13e-3bfc-49ee-80d8-8252ec2f352c"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:58:50.997052 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:50.996763 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-alertmanager-main-db" pod "f45dd13e-3bfc-49ee-80d8-8252ec2f352c" (UID: "f45dd13e-3bfc-49ee-80d8-8252ec2f352c"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:58:50.998440 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:50.998407 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-metrics-client-ca" pod "f45dd13e-3bfc-49ee-80d8-8252ec2f352c" (UID: "f45dd13e-3bfc-49ee-80d8-8252ec2f352c"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:58:50.999296 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:50.999248 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-config-out" pod "f45dd13e-3bfc-49ee-80d8-8252ec2f352c" (UID: "f45dd13e-3bfc-49ee-80d8-8252ec2f352c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:58:50.999296 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:50.999282 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-main-tls" pod "f45dd13e-3bfc-49ee-80d8-8252ec2f352c" (UID: "f45dd13e-3bfc-49ee-80d8-8252ec2f352c"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:58:50.999474 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:50.999452 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-config-volume" pod "f45dd13e-3bfc-49ee-80d8-8252ec2f352c" (UID: "f45dd13e-3bfc-49ee-80d8-8252ec2f352c"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:58:50.999869 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:50.999836 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-tls-assets" pod "f45dd13e-3bfc-49ee-80d8-8252ec2f352c" (UID: "f45dd13e-3bfc-49ee-80d8-8252ec2f352c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:58:51.000028 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:50.999999 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-kube-api-access-sprpf" pod "f45dd13e-3bfc-49ee-80d8-8252ec2f352c" (UID: "f45dd13e-3bfc-49ee-80d8-8252ec2f352c"). InnerVolumeSpecName "kube-api-access-sprpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:58:51.000139 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.000116 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-kube-rbac-proxy-web" pod "f45dd13e-3bfc-49ee-80d8-8252ec2f352c" (UID: "f45dd13e-3bfc-49ee-80d8-8252ec2f352c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:58:51.000515 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.000488 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-kube-rbac-proxy" pod "f45dd13e-3bfc-49ee-80d8-8252ec2f352c" (UID: "f45dd13e-3bfc-49ee-80d8-8252ec2f352c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:58:51.001545 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.001522 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-kube-rbac-proxy-metric" pod "f45dd13e-3bfc-49ee-80d8-8252ec2f352c" (UID: "f45dd13e-3bfc-49ee-80d8-8252ec2f352c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:58:51.003429 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.003396 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-cluster-tls-config" pod "f45dd13e-3bfc-49ee-80d8-8252ec2f352c" (UID: "f45dd13e-3bfc-49ee-80d8-8252ec2f352c"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:58:51.010692 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.010474 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-web-config" pod "f45dd13e-3bfc-49ee-80d8-8252ec2f352c" (UID: "f45dd13e-3bfc-49ee-80d8-8252ec2f352c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:58:51.096790 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.096763 7059 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:51.096790 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.096786 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sprpf\" (UniqueName: \"kubernetes.io/projected/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-kube-api-access-sprpf\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:51.096918 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.096797 7059 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:51.096918 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.096807 7059 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-config-out\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:51.096918 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.096816 7059 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:51.096918 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.096824 7059 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-tls-assets\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:51.096918 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.096833 7059 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-alertmanager-main-db\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:51.096918 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.096860 7059 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-web-config\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:51.096918 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.096869 7059 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:51.096918 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.096878 7059 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-metrics-client-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:51.096918 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.096886 7059 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-cluster-tls-config\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:51.096918 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.096894 7059 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-config-volume\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:51.096918 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.096902 7059 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f45dd13e-3bfc-49ee-80d8-8252ec2f352c-secret-alertmanager-main-tls\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:51.307449 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.307367 7059 generic.go:356] "Generic (PLEG): container finished" podID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerID="444d87bd8b809346877521d4160630c88f94461cd7f5ceff86a439d468197199" exitCode=0 Nov 04 00:58:51.307557 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.307456 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 04 00:58:51.307557 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.307452 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f45dd13e-3bfc-49ee-80d8-8252ec2f352c","Type":"ContainerDied","Data":"444d87bd8b809346877521d4160630c88f94461cd7f5ceff86a439d468197199"} Nov 04 00:58:51.307623 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.307590 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f45dd13e-3bfc-49ee-80d8-8252ec2f352c","Type":"ContainerDied","Data":"a8defe25cc00c430a9110887fea8d6dff2d0a65dfdb922885b8fdf364c92ae62"} Nov 04 00:58:51.307659 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.307624 7059 scope.go:122] "RemoveContainer" containerID="6effbe28eed4891cb772305777544e80ea635a5a35153f8cfa55897562a48642" Nov 04 00:58:51.315823 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.315803 7059 scope.go:122] "RemoveContainer" containerID="e5c8d787a70a6a5bb39ecdfa4b7346a21df9ed1b5e7546f30e0fef7669aaac9e" Nov 04 00:58:51.322785 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.322772 7059 scope.go:122] "RemoveContainer" containerID="adf1116db30959de5b988e09caba6de90c86c3d8eac0b536bc15e7d031ab3035" Nov 04 00:58:51.327778 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.327724 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 04 00:58:51.330765 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.330747 7059 scope.go:122] "RemoveContainer" containerID="444d87bd8b809346877521d4160630c88f94461cd7f5ceff86a439d468197199" Nov 04 00:58:51.331276 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.331249 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 04 00:58:51.337772 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.337753 7059 scope.go:122] "RemoveContainer" containerID="78cfe9bd056d53b1760fe77c2b25381fda1cf7e8e227c08c676986daffd8c2af" Nov 04 00:58:51.344753 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.344737 7059 scope.go:122] "RemoveContainer" containerID="b461d5ec652164a2611aa3a9967068acd7e76e9cbca9e03152786644e970d82b" Nov 04 00:58:51.351867 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.351849 7059 scope.go:122] "RemoveContainer" containerID="223d2944c74314599510a4f7c0349335629caf4e673f355fc2d608fd232009d7" Nov 04 00:58:51.360698 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.360678 7059 scope.go:122] "RemoveContainer" containerID="6effbe28eed4891cb772305777544e80ea635a5a35153f8cfa55897562a48642" Nov 04 00:58:51.361007 ip-10-0-128-129 kubenswrapper[7059]: E1104 00:58:51.360989 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6effbe28eed4891cb772305777544e80ea635a5a35153f8cfa55897562a48642\": container with ID starting with 6effbe28eed4891cb772305777544e80ea635a5a35153f8cfa55897562a48642 not found: ID does not exist" containerID="6effbe28eed4891cb772305777544e80ea635a5a35153f8cfa55897562a48642" Nov 04 00:58:51.361076 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.361017 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6effbe28eed4891cb772305777544e80ea635a5a35153f8cfa55897562a48642"} err="failed to get container status \"6effbe28eed4891cb772305777544e80ea635a5a35153f8cfa55897562a48642\": rpc error: code = NotFound desc = could not find container \"6effbe28eed4891cb772305777544e80ea635a5a35153f8cfa55897562a48642\": container with ID starting with 6effbe28eed4891cb772305777544e80ea635a5a35153f8cfa55897562a48642 not found: ID does not exist" Nov 04 00:58:51.361076 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.361049 7059 scope.go:122] "RemoveContainer" containerID="e5c8d787a70a6a5bb39ecdfa4b7346a21df9ed1b5e7546f30e0fef7669aaac9e" Nov 04 00:58:51.361289 ip-10-0-128-129 kubenswrapper[7059]: E1104 00:58:51.361269 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5c8d787a70a6a5bb39ecdfa4b7346a21df9ed1b5e7546f30e0fef7669aaac9e\": container with ID starting with e5c8d787a70a6a5bb39ecdfa4b7346a21df9ed1b5e7546f30e0fef7669aaac9e not found: ID does not exist" containerID="e5c8d787a70a6a5bb39ecdfa4b7346a21df9ed1b5e7546f30e0fef7669aaac9e" Nov 04 00:58:51.361364 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.361299 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5c8d787a70a6a5bb39ecdfa4b7346a21df9ed1b5e7546f30e0fef7669aaac9e"} err="failed to get container status \"e5c8d787a70a6a5bb39ecdfa4b7346a21df9ed1b5e7546f30e0fef7669aaac9e\": rpc error: code = NotFound desc = could not find container \"e5c8d787a70a6a5bb39ecdfa4b7346a21df9ed1b5e7546f30e0fef7669aaac9e\": container with ID starting with e5c8d787a70a6a5bb39ecdfa4b7346a21df9ed1b5e7546f30e0fef7669aaac9e not found: ID does not exist" Nov 04 00:58:51.361364 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.361323 7059 scope.go:122] "RemoveContainer" containerID="adf1116db30959de5b988e09caba6de90c86c3d8eac0b536bc15e7d031ab3035" Nov 04 00:58:51.361561 ip-10-0-128-129 kubenswrapper[7059]: E1104 00:58:51.361543 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adf1116db30959de5b988e09caba6de90c86c3d8eac0b536bc15e7d031ab3035\": container with ID starting with adf1116db30959de5b988e09caba6de90c86c3d8eac0b536bc15e7d031ab3035 not found: ID does not exist" containerID="adf1116db30959de5b988e09caba6de90c86c3d8eac0b536bc15e7d031ab3035" Nov 04 00:58:51.361630 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.361565 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adf1116db30959de5b988e09caba6de90c86c3d8eac0b536bc15e7d031ab3035"} err="failed to get container status \"adf1116db30959de5b988e09caba6de90c86c3d8eac0b536bc15e7d031ab3035\": rpc error: code = NotFound desc = could not find container \"adf1116db30959de5b988e09caba6de90c86c3d8eac0b536bc15e7d031ab3035\": container with ID starting with adf1116db30959de5b988e09caba6de90c86c3d8eac0b536bc15e7d031ab3035 not found: ID does not exist" Nov 04 00:58:51.361630 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.361584 7059 scope.go:122] "RemoveContainer" containerID="444d87bd8b809346877521d4160630c88f94461cd7f5ceff86a439d468197199" Nov 04 00:58:51.361807 ip-10-0-128-129 kubenswrapper[7059]: E1104 00:58:51.361792 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"444d87bd8b809346877521d4160630c88f94461cd7f5ceff86a439d468197199\": container with ID starting with 444d87bd8b809346877521d4160630c88f94461cd7f5ceff86a439d468197199 not found: ID does not exist" containerID="444d87bd8b809346877521d4160630c88f94461cd7f5ceff86a439d468197199" Nov 04 00:58:51.361848 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.361811 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"444d87bd8b809346877521d4160630c88f94461cd7f5ceff86a439d468197199"} err="failed to get container status \"444d87bd8b809346877521d4160630c88f94461cd7f5ceff86a439d468197199\": rpc error: code = NotFound desc = could not find container \"444d87bd8b809346877521d4160630c88f94461cd7f5ceff86a439d468197199\": container with ID starting with 444d87bd8b809346877521d4160630c88f94461cd7f5ceff86a439d468197199 not found: ID does not exist" Nov 04 00:58:51.361848 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.361824 7059 scope.go:122] "RemoveContainer" containerID="78cfe9bd056d53b1760fe77c2b25381fda1cf7e8e227c08c676986daffd8c2af" Nov 04 00:58:51.362066 ip-10-0-128-129 kubenswrapper[7059]: E1104 00:58:51.362049 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78cfe9bd056d53b1760fe77c2b25381fda1cf7e8e227c08c676986daffd8c2af\": container with ID starting with 78cfe9bd056d53b1760fe77c2b25381fda1cf7e8e227c08c676986daffd8c2af not found: ID does not exist" containerID="78cfe9bd056d53b1760fe77c2b25381fda1cf7e8e227c08c676986daffd8c2af" Nov 04 00:58:51.362136 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.362071 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78cfe9bd056d53b1760fe77c2b25381fda1cf7e8e227c08c676986daffd8c2af"} err="failed to get container status \"78cfe9bd056d53b1760fe77c2b25381fda1cf7e8e227c08c676986daffd8c2af\": rpc error: code = NotFound desc = could not find container \"78cfe9bd056d53b1760fe77c2b25381fda1cf7e8e227c08c676986daffd8c2af\": container with ID starting with 78cfe9bd056d53b1760fe77c2b25381fda1cf7e8e227c08c676986daffd8c2af not found: ID does not exist" Nov 04 00:58:51.362136 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.362084 7059 scope.go:122] "RemoveContainer" containerID="b461d5ec652164a2611aa3a9967068acd7e76e9cbca9e03152786644e970d82b" Nov 04 00:58:51.362312 ip-10-0-128-129 kubenswrapper[7059]: E1104 00:58:51.362294 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b461d5ec652164a2611aa3a9967068acd7e76e9cbca9e03152786644e970d82b\": container with ID starting with b461d5ec652164a2611aa3a9967068acd7e76e9cbca9e03152786644e970d82b not found: ID does not exist" containerID="b461d5ec652164a2611aa3a9967068acd7e76e9cbca9e03152786644e970d82b" Nov 04 00:58:51.362348 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.362317 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b461d5ec652164a2611aa3a9967068acd7e76e9cbca9e03152786644e970d82b"} err="failed to get container status \"b461d5ec652164a2611aa3a9967068acd7e76e9cbca9e03152786644e970d82b\": rpc error: code = NotFound desc = could not find container \"b461d5ec652164a2611aa3a9967068acd7e76e9cbca9e03152786644e970d82b\": container with ID starting with b461d5ec652164a2611aa3a9967068acd7e76e9cbca9e03152786644e970d82b not found: ID does not exist" Nov 04 00:58:51.362348 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.362331 7059 scope.go:122] "RemoveContainer" containerID="223d2944c74314599510a4f7c0349335629caf4e673f355fc2d608fd232009d7" Nov 04 00:58:51.362606 ip-10-0-128-129 kubenswrapper[7059]: E1104 00:58:51.362589 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"223d2944c74314599510a4f7c0349335629caf4e673f355fc2d608fd232009d7\": container with ID starting with 223d2944c74314599510a4f7c0349335629caf4e673f355fc2d608fd232009d7 not found: ID does not exist" containerID="223d2944c74314599510a4f7c0349335629caf4e673f355fc2d608fd232009d7" Nov 04 00:58:51.362645 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:51.362610 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"223d2944c74314599510a4f7c0349335629caf4e673f355fc2d608fd232009d7"} err="failed to get container status \"223d2944c74314599510a4f7c0349335629caf4e673f355fc2d608fd232009d7\": rpc error: code = NotFound desc = could not find container \"223d2944c74314599510a4f7c0349335629caf4e673f355fc2d608fd232009d7\": container with ID starting with 223d2944c74314599510a4f7c0349335629caf4e673f355fc2d608fd232009d7 not found: ID does not exist" Nov 04 00:58:53.174963 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:53.174915 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" path="/var/lib/kubelet/pods/f45dd13e-3bfc-49ee-80d8-8252ec2f352c/volumes" Nov 04 00:58:54.046524 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.046492 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 00:58:54.046998 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.046825 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerName="kube-rbac-proxy-thanos" containerID="cri-o://5d9a88f9c5dde43cfecfc6ea742cafb158898614f734f5010a0b9464fc63c044" gracePeriod=600 Nov 04 00:58:54.046998 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.046844 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerName="thanos-sidecar" containerID="cri-o://e906195f126edeb824eec62dc18fff594496021354341d88597607dba034b945" gracePeriod=600 Nov 04 00:58:54.046998 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.046853 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerName="kube-rbac-proxy-web" containerID="cri-o://080c05bf8c8df8fd6944be22f641c4387b191c82c18e2f3ae7ae692294c8c524" gracePeriod=600 Nov 04 00:58:54.046998 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.046867 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerName="config-reloader" containerID="cri-o://1c1a21c4d2257675634b3e4d9ef1aa95b4b9e7f748cf77988f3085f0a4b9bc14" gracePeriod=600 Nov 04 00:58:54.046998 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.046882 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerName="kube-rbac-proxy" containerID="cri-o://afae6491578532a4bef9c8acd31e9b111e435252399947ae1f2fd00ca59089ca" gracePeriod=600 Nov 04 00:58:54.046998 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.046822 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerName="prometheus" containerID="cri-o://ab68ecb00853ae770662fdd183ad9f573a55ee801404bdab3b6a5ece6b989e3d" gracePeriod=600 Nov 04 00:58:54.323177 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.323146 7059 generic.go:356] "Generic (PLEG): container finished" podID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerID="5d9a88f9c5dde43cfecfc6ea742cafb158898614f734f5010a0b9464fc63c044" exitCode=0 Nov 04 00:58:54.323177 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.323173 7059 generic.go:356] "Generic (PLEG): container finished" podID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerID="afae6491578532a4bef9c8acd31e9b111e435252399947ae1f2fd00ca59089ca" exitCode=0 Nov 04 00:58:54.323177 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.323181 7059 generic.go:356] "Generic (PLEG): container finished" podID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerID="080c05bf8c8df8fd6944be22f641c4387b191c82c18e2f3ae7ae692294c8c524" exitCode=0 Nov 04 00:58:54.323564 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.323176 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad0ce8d6-9cfb-4249-96ca-3866749af7f7","Type":"ContainerDied","Data":"5d9a88f9c5dde43cfecfc6ea742cafb158898614f734f5010a0b9464fc63c044"} Nov 04 00:58:54.323564 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.323248 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad0ce8d6-9cfb-4249-96ca-3866749af7f7","Type":"ContainerDied","Data":"afae6491578532a4bef9c8acd31e9b111e435252399947ae1f2fd00ca59089ca"} Nov 04 00:58:54.323564 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.323187 7059 generic.go:356] "Generic (PLEG): container finished" podID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerID="e906195f126edeb824eec62dc18fff594496021354341d88597607dba034b945" exitCode=0 Nov 04 00:58:54.323564 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.323281 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad0ce8d6-9cfb-4249-96ca-3866749af7f7","Type":"ContainerDied","Data":"080c05bf8c8df8fd6944be22f641c4387b191c82c18e2f3ae7ae692294c8c524"} Nov 04 00:58:54.323564 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.323328 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad0ce8d6-9cfb-4249-96ca-3866749af7f7","Type":"ContainerDied","Data":"e906195f126edeb824eec62dc18fff594496021354341d88597607dba034b945"} Nov 04 00:58:54.323564 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.323272 7059 generic.go:356] "Generic (PLEG): container finished" podID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerID="1c1a21c4d2257675634b3e4d9ef1aa95b4b9e7f748cf77988f3085f0a4b9bc14" exitCode=0 Nov 04 00:58:54.323564 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.323380 7059 generic.go:356] "Generic (PLEG): container finished" podID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerID="ab68ecb00853ae770662fdd183ad9f573a55ee801404bdab3b6a5ece6b989e3d" exitCode=0 Nov 04 00:58:54.323564 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.323365 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad0ce8d6-9cfb-4249-96ca-3866749af7f7","Type":"ContainerDied","Data":"1c1a21c4d2257675634b3e4d9ef1aa95b4b9e7f748cf77988f3085f0a4b9bc14"} Nov 04 00:58:54.323564 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.323466 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad0ce8d6-9cfb-4249-96ca-3866749af7f7","Type":"ContainerDied","Data":"ab68ecb00853ae770662fdd183ad9f573a55ee801404bdab3b6a5ece6b989e3d"} Nov 04 00:58:54.323564 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.323492 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad0ce8d6-9cfb-4249-96ca-3866749af7f7","Type":"ContainerDied","Data":"2d2525aeb17bd4389865536271eac885aad37b39e20e47e47da5fbc206bb7099"} Nov 04 00:58:54.323564 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.323518 7059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d2525aeb17bd4389865536271eac885aad37b39e20e47e47da5fbc206bb7099" Nov 04 00:58:54.330115 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.330095 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:54.419649 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.419617 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-thanos-prometheus-http-client-file\") pod \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " Nov 04 00:58:54.419649 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.419655 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-config\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-config\") pod \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " Nov 04 00:58:54.419884 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.419674 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-kube-rbac-proxy\") pod \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " Nov 04 00:58:54.419884 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.419710 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-configmap-kubelet-serving-ca-bundle\") pod \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " Nov 04 00:58:54.419884 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.419836 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-tls-assets\" (UniqueName: \"kubernetes.io/projected/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-tls-assets\") pod \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " Nov 04 00:58:54.420074 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.419884 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " Nov 04 00:58:54.420074 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.419958 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-rulefiles-1\") pod \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " Nov 04 00:58:54.420074 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.420002 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-metrics-client-certs\") pod \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " Nov 04 00:58:54.420074 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.420047 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " Nov 04 00:58:54.420274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.420083 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-trusted-ca-bundle\") pod \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " Nov 04 00:58:54.420274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.420117 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-configmap-kubelet-serving-ca-bundle" pod "ad0ce8d6-9cfb-4249-96ca-3866749af7f7" (UID: "ad0ce8d6-9cfb-4249-96ca-3866749af7f7"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:58:54.420274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.420124 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-rulefiles-2\") pod \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " Nov 04 00:58:54.420274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.420170 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-db\") pod \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " Nov 04 00:58:54.420274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.420201 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-configmap-serving-certs-ca-bundle\") pod \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " Nov 04 00:58:54.420274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.420225 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-config-out\" (UniqueName: \"kubernetes.io/empty-dir/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-config-out\") pod \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " Nov 04 00:58:54.420274 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.420250 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-prometheus-k8s-tls\") pod \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " Nov 04 00:58:54.420620 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.420282 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-rulefiles-0\") pod \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " Nov 04 00:58:54.420620 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.420483 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-rulefiles-2" pod "ad0ce8d6-9cfb-4249-96ca-3866749af7f7" (UID: "ad0ce8d6-9cfb-4249-96ca-3866749af7f7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:58:54.420816 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.420785 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-configmap-serving-certs-ca-bundle" pod "ad0ce8d6-9cfb-4249-96ca-3866749af7f7" (UID: "ad0ce8d6-9cfb-4249-96ca-3866749af7f7"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:58:54.420891 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.420847 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-configmap-metrics-client-ca\") pod \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " Nov 04 00:58:54.421882 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.420894 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-kube-api-access-95jzn\" (UniqueName: \"kubernetes.io/projected/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-kube-api-access-95jzn\") pod \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " Nov 04 00:58:54.421882 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.420961 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-grpc-tls\") pod \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " Nov 04 00:58:54.421882 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.420999 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-web-config\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-web-config\") pod \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\" (UID: \"ad0ce8d6-9cfb-4249-96ca-3866749af7f7\") " Nov 04 00:58:54.421882 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.421188 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-db" pod "ad0ce8d6-9cfb-4249-96ca-3866749af7f7" (UID: "ad0ce8d6-9cfb-4249-96ca-3866749af7f7"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:58:54.421882 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.421297 7059 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:54.421882 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.421317 7059 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-db\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:54.421882 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.421335 7059 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:54.421882 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.421351 7059 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:54.422438 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.422402 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-rulefiles-0" pod "ad0ce8d6-9cfb-4249-96ca-3866749af7f7" (UID: "ad0ce8d6-9cfb-4249-96ca-3866749af7f7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:58:54.422862 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.422736 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-config" pod "ad0ce8d6-9cfb-4249-96ca-3866749af7f7" (UID: "ad0ce8d6-9cfb-4249-96ca-3866749af7f7"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:58:54.423187 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.423124 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-trusted-ca-bundle" pod "ad0ce8d6-9cfb-4249-96ca-3866749af7f7" (UID: "ad0ce8d6-9cfb-4249-96ca-3866749af7f7"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:58:54.423287 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.423174 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-config-out" pod "ad0ce8d6-9cfb-4249-96ca-3866749af7f7" (UID: "ad0ce8d6-9cfb-4249-96ca-3866749af7f7"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:58:54.423632 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.423562 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-rulefiles-1" pod "ad0ce8d6-9cfb-4249-96ca-3866749af7f7" (UID: "ad0ce8d6-9cfb-4249-96ca-3866749af7f7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:58:54.423632 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.423605 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-configmap-metrics-client-ca" pod "ad0ce8d6-9cfb-4249-96ca-3866749af7f7" (UID: "ad0ce8d6-9cfb-4249-96ca-3866749af7f7"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:58:54.423771 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.423660 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-kube-rbac-proxy" pod "ad0ce8d6-9cfb-4249-96ca-3866749af7f7" (UID: "ad0ce8d6-9cfb-4249-96ca-3866749af7f7"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:58:54.423909 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.423870 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-metrics-client-certs" pod "ad0ce8d6-9cfb-4249-96ca-3866749af7f7" (UID: "ad0ce8d6-9cfb-4249-96ca-3866749af7f7"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:58:54.424082 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.424033 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-thanos-prometheus-http-client-file" pod "ad0ce8d6-9cfb-4249-96ca-3866749af7f7" (UID: "ad0ce8d6-9cfb-4249-96ca-3866749af7f7"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:58:54.425247 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.425207 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-prometheus-k8s-thanos-sidecar-tls" pod "ad0ce8d6-9cfb-4249-96ca-3866749af7f7" (UID: "ad0ce8d6-9cfb-4249-96ca-3866749af7f7"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:58:54.425519 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.425484 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-prometheus-k8s-kube-rbac-proxy-web" pod "ad0ce8d6-9cfb-4249-96ca-3866749af7f7" (UID: "ad0ce8d6-9cfb-4249-96ca-3866749af7f7"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:58:54.426238 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.426209 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-prometheus-k8s-tls" pod "ad0ce8d6-9cfb-4249-96ca-3866749af7f7" (UID: "ad0ce8d6-9cfb-4249-96ca-3866749af7f7"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:58:54.426371 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.426348 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-grpc-tls" pod "ad0ce8d6-9cfb-4249-96ca-3866749af7f7" (UID: "ad0ce8d6-9cfb-4249-96ca-3866749af7f7"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:58:54.426493 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.426471 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-tls-assets" pod "ad0ce8d6-9cfb-4249-96ca-3866749af7f7" (UID: "ad0ce8d6-9cfb-4249-96ca-3866749af7f7"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:58:54.426684 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.426648 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-kube-api-access-95jzn" pod "ad0ce8d6-9cfb-4249-96ca-3866749af7f7" (UID: "ad0ce8d6-9cfb-4249-96ca-3866749af7f7"). InnerVolumeSpecName "kube-api-access-95jzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:58:54.435420 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.435395 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-web-config" pod "ad0ce8d6-9cfb-4249-96ca-3866749af7f7" (UID: "ad0ce8d6-9cfb-4249-96ca-3866749af7f7"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:58:54.521860 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.521826 7059 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-tls-assets\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:54.521860 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.521860 7059 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:54.521997 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.521872 7059 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:54.521997 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.521881 7059 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-metrics-client-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:54.521997 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.521890 7059 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:54.521997 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.521900 7059 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:54.521997 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.521908 7059 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-config-out\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:54.521997 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.521917 7059 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:54.521997 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.521945 7059 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:54.521997 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.521953 7059 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-configmap-metrics-client-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:54.521997 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.521962 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95jzn\" (UniqueName: \"kubernetes.io/projected/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-kube-api-access-95jzn\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:54.521997 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.521970 7059 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-grpc-tls\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:54.521997 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.521979 7059 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-web-config\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:54.521997 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.521987 7059 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:54.521997 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.521996 7059 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-config\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:54.522358 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:54.522005 7059 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ad0ce8d6-9cfb-4249-96ca-3866749af7f7-secret-kube-rbac-proxy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:55.326964 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.326880 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.349454 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.349426 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 00:58:55.355157 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.355134 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 00:58:55.378512 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.378489 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 00:58:55.378810 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.378795 7059 cpu_manager.go:414] "containerMap: removing container" podUID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerName="alertmanager" Nov 04 00:58:55.378855 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.378813 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerName="alertmanager" Nov 04 00:58:55.378855 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.378829 7059 cpu_manager.go:414] "containerMap: removing container" podUID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerName="init-config-reloader" Nov 04 00:58:55.378855 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.378837 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerName="init-config-reloader" Nov 04 00:58:55.378855 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.378852 7059 cpu_manager.go:414] "containerMap: removing container" podUID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerName="thanos-sidecar" Nov 04 00:58:55.378992 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.378859 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerName="thanos-sidecar" Nov 04 00:58:55.378992 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.378873 7059 cpu_manager.go:414] "containerMap: removing container" podUID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerName="prom-label-proxy" Nov 04 00:58:55.378992 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.378877 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerName="prom-label-proxy" Nov 04 00:58:55.378992 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.378895 7059 cpu_manager.go:414] "containerMap: removing container" podUID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerName="kube-rbac-proxy-thanos" Nov 04 00:58:55.378992 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.378900 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerName="kube-rbac-proxy-thanos" Nov 04 00:58:55.378992 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.378909 7059 cpu_manager.go:414] "containerMap: removing container" podUID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerName="config-reloader" Nov 04 00:58:55.378992 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.378916 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerName="config-reloader" Nov 04 00:58:55.378992 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.378937 7059 cpu_manager.go:414] "containerMap: removing container" podUID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerName="kube-rbac-proxy-web" Nov 04 00:58:55.378992 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.378942 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerName="kube-rbac-proxy-web" Nov 04 00:58:55.378992 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.378947 7059 cpu_manager.go:414] "containerMap: removing container" podUID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerName="config-reloader" Nov 04 00:58:55.378992 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.378951 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerName="config-reloader" Nov 04 00:58:55.378992 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.378958 7059 cpu_manager.go:414] "containerMap: removing container" podUID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerName="prometheus" Nov 04 00:58:55.378992 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.378963 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerName="prometheus" Nov 04 00:58:55.378992 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.378973 7059 cpu_manager.go:414] "containerMap: removing container" podUID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerName="kube-rbac-proxy" Nov 04 00:58:55.378992 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.378980 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerName="kube-rbac-proxy" Nov 04 00:58:55.378992 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.378992 7059 cpu_manager.go:414] "containerMap: removing container" podUID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerName="kube-rbac-proxy-web" Nov 04 00:58:55.378992 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.378996 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerName="kube-rbac-proxy-web" Nov 04 00:58:55.379480 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.379010 7059 cpu_manager.go:414] "containerMap: removing container" podUID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerName="kube-rbac-proxy" Nov 04 00:58:55.379480 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.379015 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerName="kube-rbac-proxy" Nov 04 00:58:55.379480 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.379033 7059 cpu_manager.go:414] "containerMap: removing container" podUID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerName="kube-rbac-proxy-metric" Nov 04 00:58:55.379480 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.379041 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerName="kube-rbac-proxy-metric" Nov 04 00:58:55.379480 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.379051 7059 cpu_manager.go:414] "containerMap: removing container" podUID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerName="init-config-reloader" Nov 04 00:58:55.379480 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.379059 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerName="init-config-reloader" Nov 04 00:58:55.379480 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.379117 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerName="kube-rbac-proxy-metric" Nov 04 00:58:55.379480 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.379126 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerName="init-config-reloader" Nov 04 00:58:55.379480 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.379136 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerName="alertmanager" Nov 04 00:58:55.379480 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.379144 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerName="init-config-reloader" Nov 04 00:58:55.379480 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.379152 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerName="thanos-sidecar" Nov 04 00:58:55.379480 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.379160 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerName="prom-label-proxy" Nov 04 00:58:55.379480 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.379167 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerName="kube-rbac-proxy-thanos" Nov 04 00:58:55.379480 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.379176 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerName="config-reloader" Nov 04 00:58:55.379480 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.379185 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerName="kube-rbac-proxy-web" Nov 04 00:58:55.379480 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.379193 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerName="config-reloader" Nov 04 00:58:55.379480 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.379201 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerName="prometheus" Nov 04 00:58:55.379480 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.379210 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f45dd13e-3bfc-49ee-80d8-8252ec2f352c" containerName="kube-rbac-proxy" Nov 04 00:58:55.379480 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.379215 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerName="kube-rbac-proxy-web" Nov 04 00:58:55.379480 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.379225 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" containerName="kube-rbac-proxy" Nov 04 00:58:55.382525 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.382507 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.386414 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.386395 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 04 00:58:55.389203 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.389113 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 04 00:58:55.389203 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.389179 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 04 00:58:55.389203 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.389201 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 04 00:58:55.389353 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.389263 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 04 00:58:55.389353 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.389272 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 04 00:58:55.389545 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.389506 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 04 00:58:55.389907 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.389661 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 04 00:58:55.389907 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.389707 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 04 00:58:55.389907 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.389711 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 04 00:58:55.389907 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.389743 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wzbxc\"" Nov 04 00:58:55.389907 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.389886 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 04 00:58:55.391205 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.391049 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9outkgf7grlms\"" Nov 04 00:58:55.391205 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.391100 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 04 00:58:55.391388 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.391267 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 04 00:58:55.391535 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.391518 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 04 00:58:55.397337 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.397311 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 00:58:55.530775 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.530753 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b7c4ca5-64c2-43d7-9963-90b35205391b-config-out\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.530910 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.530782 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b7c4ca5-64c2-43d7-9963-90b35205391b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.530910 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.530804 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0b7c4ca5-64c2-43d7-9963-90b35205391b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.530910 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.530867 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0b7c4ca5-64c2-43d7-9963-90b35205391b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.530910 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.530891 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b7c4ca5-64c2-43d7-9963-90b35205391b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.531063 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.530959 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0b7c4ca5-64c2-43d7-9963-90b35205391b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.531063 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.530999 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b7c4ca5-64c2-43d7-9963-90b35205391b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.531122 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.531064 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0b7c4ca5-64c2-43d7-9963-90b35205391b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.531122 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.531096 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mwjb\" (UniqueName: \"kubernetes.io/projected/0b7c4ca5-64c2-43d7-9963-90b35205391b-kube-api-access-4mwjb\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.531186 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.531170 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0b7c4ca5-64c2-43d7-9963-90b35205391b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.531220 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.531203 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b7c4ca5-64c2-43d7-9963-90b35205391b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.531252 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.531220 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b7c4ca5-64c2-43d7-9963-90b35205391b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.531252 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.531236 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0b7c4ca5-64c2-43d7-9963-90b35205391b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.531313 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.531260 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b7c4ca5-64c2-43d7-9963-90b35205391b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.531313 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.531283 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0b7c4ca5-64c2-43d7-9963-90b35205391b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.531380 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.531346 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0b7c4ca5-64c2-43d7-9963-90b35205391b-config\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.531409 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.531379 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b7c4ca5-64c2-43d7-9963-90b35205391b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.531409 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.531401 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0b7c4ca5-64c2-43d7-9963-90b35205391b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.531468 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.531440 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0b7c4ca5-64c2-43d7-9963-90b35205391b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.531468 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.531458 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b7c4ca5-64c2-43d7-9963-90b35205391b-web-config\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.632360 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.632315 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b7c4ca5-64c2-43d7-9963-90b35205391b-config-out\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.632360 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.632339 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b7c4ca5-64c2-43d7-9963-90b35205391b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.632360 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.632358 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0b7c4ca5-64c2-43d7-9963-90b35205391b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.632495 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.632375 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0b7c4ca5-64c2-43d7-9963-90b35205391b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.632495 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.632394 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b7c4ca5-64c2-43d7-9963-90b35205391b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.632495 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.632409 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0b7c4ca5-64c2-43d7-9963-90b35205391b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.632495 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.632425 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b7c4ca5-64c2-43d7-9963-90b35205391b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.632685 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.632635 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0b7c4ca5-64c2-43d7-9963-90b35205391b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.632685 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.632662 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mwjb\" (UniqueName: \"kubernetes.io/projected/0b7c4ca5-64c2-43d7-9963-90b35205391b-kube-api-access-4mwjb\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.632793 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.632691 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0b7c4ca5-64c2-43d7-9963-90b35205391b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.632793 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.632724 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b7c4ca5-64c2-43d7-9963-90b35205391b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.632793 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.632751 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b7c4ca5-64c2-43d7-9963-90b35205391b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.632793 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.632773 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0b7c4ca5-64c2-43d7-9963-90b35205391b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.633024 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.632777 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0b7c4ca5-64c2-43d7-9963-90b35205391b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.633024 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.632790 7059 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 04 00:58:55.633024 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.632849 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b7c4ca5-64c2-43d7-9963-90b35205391b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.633024 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.632882 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0b7c4ca5-64c2-43d7-9963-90b35205391b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.633024 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.632913 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0b7c4ca5-64c2-43d7-9963-90b35205391b-config\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.633024 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.632969 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b7c4ca5-64c2-43d7-9963-90b35205391b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.633024 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.632998 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0b7c4ca5-64c2-43d7-9963-90b35205391b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.633525 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.633049 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0b7c4ca5-64c2-43d7-9963-90b35205391b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.633525 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.633054 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0b7c4ca5-64c2-43d7-9963-90b35205391b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.633525 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.633086 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b7c4ca5-64c2-43d7-9963-90b35205391b-web-config\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.633525 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.633237 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b7c4ca5-64c2-43d7-9963-90b35205391b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.633525 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.633247 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b7c4ca5-64c2-43d7-9963-90b35205391b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.633789 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.633688 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b7c4ca5-64c2-43d7-9963-90b35205391b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.634194 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.634170 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0b7c4ca5-64c2-43d7-9963-90b35205391b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.637881 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.635440 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b7c4ca5-64c2-43d7-9963-90b35205391b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.637881 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.637802 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0b7c4ca5-64c2-43d7-9963-90b35205391b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.638057 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.637996 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b7c4ca5-64c2-43d7-9963-90b35205391b-config-out\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.638834 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.638395 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b7c4ca5-64c2-43d7-9963-90b35205391b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.638834 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.638546 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b7c4ca5-64c2-43d7-9963-90b35205391b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.638834 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.638750 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b7c4ca5-64c2-43d7-9963-90b35205391b-web-config\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.639084 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.639056 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0b7c4ca5-64c2-43d7-9963-90b35205391b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.639265 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.639243 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0b7c4ca5-64c2-43d7-9963-90b35205391b-config\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.639397 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.639291 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b7c4ca5-64c2-43d7-9963-90b35205391b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.639647 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.639559 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0b7c4ca5-64c2-43d7-9963-90b35205391b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.639714 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.639695 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0b7c4ca5-64c2-43d7-9963-90b35205391b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.641448 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.641419 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0b7c4ca5-64c2-43d7-9963-90b35205391b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.642551 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.642521 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mwjb\" (UniqueName: \"kubernetes.io/projected/0b7c4ca5-64c2-43d7-9963-90b35205391b-kube-api-access-4mwjb\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.644281 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.644262 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0b7c4ca5-64c2-43d7-9963-90b35205391b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0b7c4ca5-64c2-43d7-9963-90b35205391b\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.702470 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.702440 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:58:55.980580 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:55.980551 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 04 00:58:55.983723 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:55.983696 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b7c4ca5_64c2_43d7_9963_90b35205391b.slice/crio-a870795520059502c50f0514ee3354d501f8ff909f1928bfc0d613651eb57ed9 WatchSource:0}: Error finding container a870795520059502c50f0514ee3354d501f8ff909f1928bfc0d613651eb57ed9: Status 404 returned error can't find the container with id a870795520059502c50f0514ee3354d501f8ff909f1928bfc0d613651eb57ed9 Nov 04 00:58:56.332042 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:56.331965 7059 generic.go:356] "Generic (PLEG): container finished" podID="0b7c4ca5-64c2-43d7-9963-90b35205391b" containerID="c441a8ae6d3f2a14c0f4b4c95ac9955b94e322d22873a50db5abdf0c6a6d4ba2" exitCode=0 Nov 04 00:58:56.332042 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:56.332019 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b7c4ca5-64c2-43d7-9963-90b35205391b","Type":"ContainerDied","Data":"c441a8ae6d3f2a14c0f4b4c95ac9955b94e322d22873a50db5abdf0c6a6d4ba2"} Nov 04 00:58:56.332514 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:56.332072 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b7c4ca5-64c2-43d7-9963-90b35205391b","Type":"ContainerStarted","Data":"a870795520059502c50f0514ee3354d501f8ff909f1928bfc0d613651eb57ed9"} Nov 04 00:58:57.174709 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.174675 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad0ce8d6-9cfb-4249-96ca-3866749af7f7" path="/var/lib/kubelet/pods/ad0ce8d6-9cfb-4249-96ca-3866749af7f7/volumes" Nov 04 00:58:57.207539 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.207513 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-fb659d84-hqndc"] Nov 04 00:58:57.209703 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.209687 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fb659d84-hqndc" Nov 04 00:58:57.225400 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.225363 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fb659d84-hqndc"] Nov 04 00:58:57.340197 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.340164 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b7c4ca5-64c2-43d7-9963-90b35205391b","Type":"ContainerStarted","Data":"ec37da65813ab7a3b85de38fd17ff896d83588fdb9a299bae5d7de51b61fae2f"} Nov 04 00:58:57.340488 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.340287 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b7c4ca5-64c2-43d7-9963-90b35205391b","Type":"ContainerStarted","Data":"ff343efb2672d0f20a7fc2b006f30e7692bca1d93d76c40fff6a9b7e1a45ff18"} Nov 04 00:58:57.340488 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.340330 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b7c4ca5-64c2-43d7-9963-90b35205391b","Type":"ContainerStarted","Data":"13540dbd88aa424edc5424f52a593222f2360bc24017bcd369af144fcdeacc1d"} Nov 04 00:58:57.340488 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.340374 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b7c4ca5-64c2-43d7-9963-90b35205391b","Type":"ContainerStarted","Data":"17d54e2ffe5ef89e70faff83e64db1037d0a9deb65480fb7d207999bdade0b29"} Nov 04 00:58:57.340488 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.340411 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b7c4ca5-64c2-43d7-9963-90b35205391b","Type":"ContainerStarted","Data":"e5779692bb522c5e7bee75eb910991f31b72a51dc775631117a478d94a9ffe6f"} Nov 04 00:58:57.340488 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.340445 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b7c4ca5-64c2-43d7-9963-90b35205391b","Type":"ContainerStarted","Data":"7fc68013e2b6d63a6bb4efd8a5a2f85b154e0c0e64fcd3cb2ed9777fcffe76b1"} Nov 04 00:58:57.350100 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.350072 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e8e30c89-6de5-458e-a105-6e37373450cb-oauth-serving-cert\") pod \"console-fb659d84-hqndc\" (UID: \"e8e30c89-6de5-458e-a105-6e37373450cb\") " pod="openshift-console/console-fb659d84-hqndc" Nov 04 00:58:57.350195 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.350124 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e8e30c89-6de5-458e-a105-6e37373450cb-console-config\") pod \"console-fb659d84-hqndc\" (UID: \"e8e30c89-6de5-458e-a105-6e37373450cb\") " pod="openshift-console/console-fb659d84-hqndc" Nov 04 00:58:57.350234 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.350192 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8e30c89-6de5-458e-a105-6e37373450cb-tmp\") pod \"console-fb659d84-hqndc\" (UID: \"e8e30c89-6de5-458e-a105-6e37373450cb\") " pod="openshift-console/console-fb659d84-hqndc" Nov 04 00:58:57.350234 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.350222 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e8e30c89-6de5-458e-a105-6e37373450cb-console-oauth-config\") pod \"console-fb659d84-hqndc\" (UID: \"e8e30c89-6de5-458e-a105-6e37373450cb\") " pod="openshift-console/console-fb659d84-hqndc" Nov 04 00:58:57.350375 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.350261 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e8e30c89-6de5-458e-a105-6e37373450cb-service-ca\") pod \"console-fb659d84-hqndc\" (UID: \"e8e30c89-6de5-458e-a105-6e37373450cb\") " pod="openshift-console/console-fb659d84-hqndc" Nov 04 00:58:57.350425 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.350402 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e8e30c89-6de5-458e-a105-6e37373450cb-console-serving-cert\") pod \"console-fb659d84-hqndc\" (UID: \"e8e30c89-6de5-458e-a105-6e37373450cb\") " pod="openshift-console/console-fb659d84-hqndc" Nov 04 00:58:57.350473 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.350455 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8e30c89-6de5-458e-a105-6e37373450cb-trusted-ca-bundle\") pod \"console-fb659d84-hqndc\" (UID: \"e8e30c89-6de5-458e-a105-6e37373450cb\") " pod="openshift-console/console-fb659d84-hqndc" Nov 04 00:58:57.350504 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.350486 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p554\" (UniqueName: \"kubernetes.io/projected/e8e30c89-6de5-458e-a105-6e37373450cb-kube-api-access-6p554\") pod \"console-fb659d84-hqndc\" (UID: \"e8e30c89-6de5-458e-a105-6e37373450cb\") " pod="openshift-console/console-fb659d84-hqndc" Nov 04 00:58:57.371084 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.371033 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.3517459880000002 podStartE2EDuration="2.36897464s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.228652ms" isStatefulPod=false podCreationTimestamp="2026-11-04 00:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:57.367356299 +0000 UTC m=+22.309205201" watchObservedRunningTime="2026-11-04 00:58:57.36897464 +0000 UTC m=+22.310823544" Nov 04 00:58:57.450913 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.450850 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e8e30c89-6de5-458e-a105-6e37373450cb-service-ca\") pod \"console-fb659d84-hqndc\" (UID: \"e8e30c89-6de5-458e-a105-6e37373450cb\") " pod="openshift-console/console-fb659d84-hqndc" Nov 04 00:58:57.451065 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.451004 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e8e30c89-6de5-458e-a105-6e37373450cb-console-serving-cert\") pod \"console-fb659d84-hqndc\" (UID: \"e8e30c89-6de5-458e-a105-6e37373450cb\") " pod="openshift-console/console-fb659d84-hqndc" Nov 04 00:58:57.451138 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.451081 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8e30c89-6de5-458e-a105-6e37373450cb-trusted-ca-bundle\") pod \"console-fb659d84-hqndc\" (UID: \"e8e30c89-6de5-458e-a105-6e37373450cb\") " pod="openshift-console/console-fb659d84-hqndc" Nov 04 00:58:57.451138 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.451124 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p554\" (UniqueName: \"kubernetes.io/projected/e8e30c89-6de5-458e-a105-6e37373450cb-kube-api-access-6p554\") pod \"console-fb659d84-hqndc\" (UID: \"e8e30c89-6de5-458e-a105-6e37373450cb\") " pod="openshift-console/console-fb659d84-hqndc" Nov 04 00:58:57.451651 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.451245 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e8e30c89-6de5-458e-a105-6e37373450cb-oauth-serving-cert\") pod \"console-fb659d84-hqndc\" (UID: \"e8e30c89-6de5-458e-a105-6e37373450cb\") " pod="openshift-console/console-fb659d84-hqndc" Nov 04 00:58:57.451651 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.451547 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e8e30c89-6de5-458e-a105-6e37373450cb-console-config\") pod \"console-fb659d84-hqndc\" (UID: \"e8e30c89-6de5-458e-a105-6e37373450cb\") " pod="openshift-console/console-fb659d84-hqndc" Nov 04 00:58:57.451651 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.451603 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8e30c89-6de5-458e-a105-6e37373450cb-tmp\") pod \"console-fb659d84-hqndc\" (UID: \"e8e30c89-6de5-458e-a105-6e37373450cb\") " pod="openshift-console/console-fb659d84-hqndc" Nov 04 00:58:57.451651 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.451629 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e8e30c89-6de5-458e-a105-6e37373450cb-console-oauth-config\") pod \"console-fb659d84-hqndc\" (UID: \"e8e30c89-6de5-458e-a105-6e37373450cb\") " pod="openshift-console/console-fb659d84-hqndc" Nov 04 00:58:57.452119 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.451673 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e8e30c89-6de5-458e-a105-6e37373450cb-service-ca\") pod \"console-fb659d84-hqndc\" (UID: \"e8e30c89-6de5-458e-a105-6e37373450cb\") " pod="openshift-console/console-fb659d84-hqndc" Nov 04 00:58:57.452119 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.452024 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8e30c89-6de5-458e-a105-6e37373450cb-tmp\") pod \"console-fb659d84-hqndc\" (UID: \"e8e30c89-6de5-458e-a105-6e37373450cb\") " pod="openshift-console/console-fb659d84-hqndc" Nov 04 00:58:57.452228 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.452150 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8e30c89-6de5-458e-a105-6e37373450cb-trusted-ca-bundle\") pod \"console-fb659d84-hqndc\" (UID: \"e8e30c89-6de5-458e-a105-6e37373450cb\") " pod="openshift-console/console-fb659d84-hqndc" Nov 04 00:58:57.452446 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.452274 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e8e30c89-6de5-458e-a105-6e37373450cb-console-config\") pod \"console-fb659d84-hqndc\" (UID: \"e8e30c89-6de5-458e-a105-6e37373450cb\") " pod="openshift-console/console-fb659d84-hqndc" Nov 04 00:58:57.452527 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.452504 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e8e30c89-6de5-458e-a105-6e37373450cb-oauth-serving-cert\") pod \"console-fb659d84-hqndc\" (UID: \"e8e30c89-6de5-458e-a105-6e37373450cb\") " pod="openshift-console/console-fb659d84-hqndc" Nov 04 00:58:57.454093 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.454070 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e8e30c89-6de5-458e-a105-6e37373450cb-console-oauth-config\") pod \"console-fb659d84-hqndc\" (UID: \"e8e30c89-6de5-458e-a105-6e37373450cb\") " pod="openshift-console/console-fb659d84-hqndc" Nov 04 00:58:57.454236 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.454220 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e8e30c89-6de5-458e-a105-6e37373450cb-console-serving-cert\") pod \"console-fb659d84-hqndc\" (UID: \"e8e30c89-6de5-458e-a105-6e37373450cb\") " pod="openshift-console/console-fb659d84-hqndc" Nov 04 00:58:57.459757 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.459740 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p554\" (UniqueName: \"kubernetes.io/projected/e8e30c89-6de5-458e-a105-6e37373450cb-kube-api-access-6p554\") pod \"console-fb659d84-hqndc\" (UID: \"e8e30c89-6de5-458e-a105-6e37373450cb\") " pod="openshift-console/console-fb659d84-hqndc" Nov 04 00:58:57.520232 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.520214 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fb659d84-hqndc" Nov 04 00:58:57.790449 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:57.790386 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fb659d84-hqndc"] Nov 04 00:58:57.793529 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:58:57.793505 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e30c89_6de5_458e_a105_6e37373450cb.slice/crio-75eff6f81a4aabb14b9835e27f1abdae2b5a2c64679f6f4c95ccb9570c6c6440 WatchSource:0}: Error finding container 75eff6f81a4aabb14b9835e27f1abdae2b5a2c64679f6f4c95ccb9570c6c6440: Status 404 returned error can't find the container with id 75eff6f81a4aabb14b9835e27f1abdae2b5a2c64679f6f4c95ccb9570c6c6440 Nov 04 00:58:58.346322 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:58.346289 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fb659d84-hqndc" event={"ID":"e8e30c89-6de5-458e-a105-6e37373450cb","Type":"ContainerStarted","Data":"0424946a741e63bd8d67407b2f726ed801db70a16e09e123ca5916ed667bf81c"} Nov 04 00:58:58.346691 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:58.346338 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fb659d84-hqndc" event={"ID":"e8e30c89-6de5-458e-a105-6e37373450cb","Type":"ContainerStarted","Data":"75eff6f81a4aabb14b9835e27f1abdae2b5a2c64679f6f4c95ccb9570c6c6440"} Nov 04 00:58:58.365569 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:58.365522 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-fb659d84-hqndc" podStartSLOduration=1.365509062 podStartE2EDuration="1.365509062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:58:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:58.364711261 +0000 UTC m=+23.306560164" watchObservedRunningTime="2026-11-04 00:58:58.365509062 +0000 UTC m=+23.307357965" Nov 04 00:58:59.433201 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:58:59.433175 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nb955" Nov 04 00:59:00.703552 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:00.703522 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:59:05.049333 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:05.049300 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xwbdv" Nov 04 00:59:06.478301 ip-10-0-128-129 kubenswrapper[7059]: E1104 00:59:06.478262 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 00:59:07.521217 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:07.521192 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-fb659d84-hqndc" Nov 04 00:59:07.521670 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:07.521239 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-fb659d84-hqndc" Nov 04 00:59:07.526191 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:07.526172 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-fb659d84-hqndc" Nov 04 00:59:08.387656 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:08.387629 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-fb659d84-hqndc" Nov 04 00:59:08.431162 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:08.431129 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69dd78b66f-clp8w"] Nov 04 00:59:10.149471 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:10.149443 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-289kb" Nov 04 00:59:28.083661 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:28.083624 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:59:28.099406 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:28.099375 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:59:28.470629 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:28.470602 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 04 00:59:33.456834 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:33.456763 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-69dd78b66f-clp8w" podUID="34aecbea-ff97-4f23-8336-326316720a2f" containerName="console" containerID="cri-o://a7efcc09b9005f34d05af2ff13cd52b5d3531141b639d5556415b2a0543ca983" gracePeriod=15 Nov 04 00:59:33.713603 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:33.713553 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:59:33.825392 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:33.825366 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-service-ca\" (UniqueName: \"kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-service-ca\") pod \"34aecbea-ff97-4f23-8336-326316720a2f\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " Nov 04 00:59:33.825542 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:33.825399 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-oauth-serving-cert\") pod \"34aecbea-ff97-4f23-8336-326316720a2f\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " Nov 04 00:59:33.825542 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:33.825437 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34aecbea-ff97-4f23-8336-326316720a2f-kube-api-access-79nvt\" (UniqueName: \"kubernetes.io/projected/34aecbea-ff97-4f23-8336-326316720a2f-kube-api-access-79nvt\") pod \"34aecbea-ff97-4f23-8336-326316720a2f\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " Nov 04 00:59:33.825651 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:33.825563 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34aecbea-ff97-4f23-8336-326316720a2f-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/34aecbea-ff97-4f23-8336-326316720a2f-console-oauth-config\") pod \"34aecbea-ff97-4f23-8336-326316720a2f\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " Nov 04 00:59:33.825694 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:33.825647 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-trusted-ca-bundle\") pod \"34aecbea-ff97-4f23-8336-326316720a2f\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " Nov 04 00:59:33.825741 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:33.825695 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34aecbea-ff97-4f23-8336-326316720a2f-tmp\" (UniqueName: \"kubernetes.io/empty-dir/34aecbea-ff97-4f23-8336-326316720a2f-tmp\") pod \"34aecbea-ff97-4f23-8336-326316720a2f\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " Nov 04 00:59:33.825741 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:33.825733 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-console-config\" (UniqueName: \"kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-console-config\") pod \"34aecbea-ff97-4f23-8336-326316720a2f\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " Nov 04 00:59:33.825908 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:33.825792 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-service-ca" pod "34aecbea-ff97-4f23-8336-326316720a2f" (UID: "34aecbea-ff97-4f23-8336-326316720a2f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:59:33.825908 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:33.825801 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34aecbea-ff97-4f23-8336-326316720a2f-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/34aecbea-ff97-4f23-8336-326316720a2f-console-serving-cert\") pod \"34aecbea-ff97-4f23-8336-326316720a2f\" (UID: \"34aecbea-ff97-4f23-8336-326316720a2f\") " Nov 04 00:59:33.826045 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:33.825992 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-oauth-serving-cert" pod "34aecbea-ff97-4f23-8336-326316720a2f" (UID: "34aecbea-ff97-4f23-8336-326316720a2f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:59:33.826100 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:33.826049 7059 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-service-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:59:33.826100 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:33.826061 7059 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-oauth-serving-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:59:33.826233 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:33.826206 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-trusted-ca-bundle" pod "34aecbea-ff97-4f23-8336-326316720a2f" (UID: "34aecbea-ff97-4f23-8336-326316720a2f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:59:33.826431 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:33.826404 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-console-config" pod "34aecbea-ff97-4f23-8336-326316720a2f" (UID: "34aecbea-ff97-4f23-8336-326316720a2f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:59:33.826535 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:33.826462 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34aecbea-ff97-4f23-8336-326316720a2f-tmp" pod "34aecbea-ff97-4f23-8336-326316720a2f" (UID: "34aecbea-ff97-4f23-8336-326316720a2f"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:59:33.827959 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:33.827897 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34aecbea-ff97-4f23-8336-326316720a2f-console-oauth-config" pod "34aecbea-ff97-4f23-8336-326316720a2f" (UID: "34aecbea-ff97-4f23-8336-326316720a2f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:59:33.827959 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:33.827950 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34aecbea-ff97-4f23-8336-326316720a2f-kube-api-access-79nvt" pod "34aecbea-ff97-4f23-8336-326316720a2f" (UID: "34aecbea-ff97-4f23-8336-326316720a2f"). InnerVolumeSpecName "kube-api-access-79nvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:59:33.828166 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:33.828151 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34aecbea-ff97-4f23-8336-326316720a2f-console-serving-cert" pod "34aecbea-ff97-4f23-8336-326316720a2f" (UID: "34aecbea-ff97-4f23-8336-326316720a2f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:59:33.926721 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:33.926693 7059 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-trusted-ca-bundle\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:59:33.926808 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:33.926725 7059 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34aecbea-ff97-4f23-8336-326316720a2f-tmp\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:59:33.926808 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:33.926736 7059 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/34aecbea-ff97-4f23-8336-326316720a2f-console-config\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:59:33.926808 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:33.926745 7059 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/34aecbea-ff97-4f23-8336-326316720a2f-console-serving-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:59:33.926808 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:33.926754 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79nvt\" (UniqueName: \"kubernetes.io/projected/34aecbea-ff97-4f23-8336-326316720a2f-kube-api-access-79nvt\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:59:33.926808 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:33.926763 7059 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/34aecbea-ff97-4f23-8336-326316720a2f-console-oauth-config\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:59:34.161433 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.161355 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-65dcbb7874-lwgl8"] Nov 04 00:59:34.161759 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.161739 7059 cpu_manager.go:414] "containerMap: removing container" podUID="34aecbea-ff97-4f23-8336-326316720a2f" containerName="console" Nov 04 00:59:34.161858 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.161761 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34aecbea-ff97-4f23-8336-326316720a2f" containerName="console" Nov 04 00:59:34.161858 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.161852 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="34aecbea-ff97-4f23-8336-326316720a2f" containerName="console" Nov 04 00:59:34.163785 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.163763 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65dcbb7874-lwgl8" Nov 04 00:59:34.174536 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.174515 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65dcbb7874-lwgl8"] Nov 04 00:59:34.229308 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.229279 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df880d85-2da0-40fc-a106-22af2ad25cd7-console-oauth-config\") pod \"console-65dcbb7874-lwgl8\" (UID: \"df880d85-2da0-40fc-a106-22af2ad25cd7\") " pod="openshift-console/console-65dcbb7874-lwgl8" Nov 04 00:59:34.229419 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.229320 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df880d85-2da0-40fc-a106-22af2ad25cd7-console-config\") pod \"console-65dcbb7874-lwgl8\" (UID: \"df880d85-2da0-40fc-a106-22af2ad25cd7\") " pod="openshift-console/console-65dcbb7874-lwgl8" Nov 04 00:59:34.229419 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.229337 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df880d85-2da0-40fc-a106-22af2ad25cd7-oauth-serving-cert\") pod \"console-65dcbb7874-lwgl8\" (UID: \"df880d85-2da0-40fc-a106-22af2ad25cd7\") " pod="openshift-console/console-65dcbb7874-lwgl8" Nov 04 00:59:34.229501 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.229443 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df880d85-2da0-40fc-a106-22af2ad25cd7-trusted-ca-bundle\") pod \"console-65dcbb7874-lwgl8\" (UID: \"df880d85-2da0-40fc-a106-22af2ad25cd7\") " pod="openshift-console/console-65dcbb7874-lwgl8" Nov 04 00:59:34.229501 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.229472 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df880d85-2da0-40fc-a106-22af2ad25cd7-console-serving-cert\") pod \"console-65dcbb7874-lwgl8\" (UID: \"df880d85-2da0-40fc-a106-22af2ad25cd7\") " pod="openshift-console/console-65dcbb7874-lwgl8" Nov 04 00:59:34.229501 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.229497 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df880d85-2da0-40fc-a106-22af2ad25cd7-tmp\") pod \"console-65dcbb7874-lwgl8\" (UID: \"df880d85-2da0-40fc-a106-22af2ad25cd7\") " pod="openshift-console/console-65dcbb7874-lwgl8" Nov 04 00:59:34.229606 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.229515 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6kqb\" (UniqueName: \"kubernetes.io/projected/df880d85-2da0-40fc-a106-22af2ad25cd7-kube-api-access-z6kqb\") pod \"console-65dcbb7874-lwgl8\" (UID: \"df880d85-2da0-40fc-a106-22af2ad25cd7\") " pod="openshift-console/console-65dcbb7874-lwgl8" Nov 04 00:59:34.229643 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.229629 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df880d85-2da0-40fc-a106-22af2ad25cd7-service-ca\") pod \"console-65dcbb7874-lwgl8\" (UID: \"df880d85-2da0-40fc-a106-22af2ad25cd7\") " pod="openshift-console/console-65dcbb7874-lwgl8" Nov 04 00:59:34.330707 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.330681 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df880d85-2da0-40fc-a106-22af2ad25cd7-trusted-ca-bundle\") pod \"console-65dcbb7874-lwgl8\" (UID: \"df880d85-2da0-40fc-a106-22af2ad25cd7\") " pod="openshift-console/console-65dcbb7874-lwgl8" Nov 04 00:59:34.330823 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.330711 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df880d85-2da0-40fc-a106-22af2ad25cd7-console-serving-cert\") pod \"console-65dcbb7874-lwgl8\" (UID: \"df880d85-2da0-40fc-a106-22af2ad25cd7\") " pod="openshift-console/console-65dcbb7874-lwgl8" Nov 04 00:59:34.330823 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.330743 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df880d85-2da0-40fc-a106-22af2ad25cd7-tmp\") pod \"console-65dcbb7874-lwgl8\" (UID: \"df880d85-2da0-40fc-a106-22af2ad25cd7\") " pod="openshift-console/console-65dcbb7874-lwgl8" Nov 04 00:59:34.330956 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.330917 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6kqb\" (UniqueName: \"kubernetes.io/projected/df880d85-2da0-40fc-a106-22af2ad25cd7-kube-api-access-z6kqb\") pod \"console-65dcbb7874-lwgl8\" (UID: \"df880d85-2da0-40fc-a106-22af2ad25cd7\") " pod="openshift-console/console-65dcbb7874-lwgl8" Nov 04 00:59:34.331010 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.330979 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df880d85-2da0-40fc-a106-22af2ad25cd7-service-ca\") pod \"console-65dcbb7874-lwgl8\" (UID: \"df880d85-2da0-40fc-a106-22af2ad25cd7\") " pod="openshift-console/console-65dcbb7874-lwgl8" Nov 04 00:59:34.331010 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.330997 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df880d85-2da0-40fc-a106-22af2ad25cd7-console-oauth-config\") pod \"console-65dcbb7874-lwgl8\" (UID: \"df880d85-2da0-40fc-a106-22af2ad25cd7\") " pod="openshift-console/console-65dcbb7874-lwgl8" Nov 04 00:59:34.331118 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.331024 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df880d85-2da0-40fc-a106-22af2ad25cd7-console-config\") pod \"console-65dcbb7874-lwgl8\" (UID: \"df880d85-2da0-40fc-a106-22af2ad25cd7\") " pod="openshift-console/console-65dcbb7874-lwgl8" Nov 04 00:59:34.331118 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.331039 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df880d85-2da0-40fc-a106-22af2ad25cd7-oauth-serving-cert\") pod \"console-65dcbb7874-lwgl8\" (UID: \"df880d85-2da0-40fc-a106-22af2ad25cd7\") " pod="openshift-console/console-65dcbb7874-lwgl8" Nov 04 00:59:34.331310 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.331289 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df880d85-2da0-40fc-a106-22af2ad25cd7-tmp\") pod \"console-65dcbb7874-lwgl8\" (UID: \"df880d85-2da0-40fc-a106-22af2ad25cd7\") " pod="openshift-console/console-65dcbb7874-lwgl8" Nov 04 00:59:34.331756 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.331573 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df880d85-2da0-40fc-a106-22af2ad25cd7-trusted-ca-bundle\") pod \"console-65dcbb7874-lwgl8\" (UID: \"df880d85-2da0-40fc-a106-22af2ad25cd7\") " pod="openshift-console/console-65dcbb7874-lwgl8" Nov 04 00:59:34.331756 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.331717 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df880d85-2da0-40fc-a106-22af2ad25cd7-oauth-serving-cert\") pod \"console-65dcbb7874-lwgl8\" (UID: \"df880d85-2da0-40fc-a106-22af2ad25cd7\") " pod="openshift-console/console-65dcbb7874-lwgl8" Nov 04 00:59:34.331756 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.331717 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df880d85-2da0-40fc-a106-22af2ad25cd7-console-config\") pod \"console-65dcbb7874-lwgl8\" (UID: \"df880d85-2da0-40fc-a106-22af2ad25cd7\") " pod="openshift-console/console-65dcbb7874-lwgl8" Nov 04 00:59:34.332081 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.332057 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df880d85-2da0-40fc-a106-22af2ad25cd7-service-ca\") pod \"console-65dcbb7874-lwgl8\" (UID: \"df880d85-2da0-40fc-a106-22af2ad25cd7\") " pod="openshift-console/console-65dcbb7874-lwgl8" Nov 04 00:59:34.333478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.333455 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df880d85-2da0-40fc-a106-22af2ad25cd7-console-serving-cert\") pod \"console-65dcbb7874-lwgl8\" (UID: \"df880d85-2da0-40fc-a106-22af2ad25cd7\") " pod="openshift-console/console-65dcbb7874-lwgl8" Nov 04 00:59:34.333478 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.333473 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df880d85-2da0-40fc-a106-22af2ad25cd7-console-oauth-config\") pod \"console-65dcbb7874-lwgl8\" (UID: \"df880d85-2da0-40fc-a106-22af2ad25cd7\") " pod="openshift-console/console-65dcbb7874-lwgl8" Nov 04 00:59:34.338865 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.338848 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6kqb\" (UniqueName: \"kubernetes.io/projected/df880d85-2da0-40fc-a106-22af2ad25cd7-kube-api-access-z6kqb\") pod \"console-65dcbb7874-lwgl8\" (UID: \"df880d85-2da0-40fc-a106-22af2ad25cd7\") " pod="openshift-console/console-65dcbb7874-lwgl8" Nov 04 00:59:34.473327 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.473305 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65dcbb7874-lwgl8" Nov 04 00:59:34.477715 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.477689 7059 generic.go:356] "Generic (PLEG): container finished" podID="34aecbea-ff97-4f23-8336-326316720a2f" containerID="a7efcc09b9005f34d05af2ff13cd52b5d3531141b639d5556415b2a0543ca983" exitCode=0 Nov 04 00:59:34.477812 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.477738 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69dd78b66f-clp8w" event={"ID":"34aecbea-ff97-4f23-8336-326316720a2f","Type":"ContainerDied","Data":"a7efcc09b9005f34d05af2ff13cd52b5d3531141b639d5556415b2a0543ca983"} Nov 04 00:59:34.477812 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.477755 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69dd78b66f-clp8w" Nov 04 00:59:34.477812 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.477808 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69dd78b66f-clp8w" event={"ID":"34aecbea-ff97-4f23-8336-326316720a2f","Type":"ContainerDied","Data":"542986989c9fe1332f4a001b80ae9f7c3a566c325acb0350072842372c21b519"} Nov 04 00:59:34.478001 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.477828 7059 scope.go:122] "RemoveContainer" containerID="a7efcc09b9005f34d05af2ff13cd52b5d3531141b639d5556415b2a0543ca983" Nov 04 00:59:34.496104 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.496079 7059 scope.go:122] "RemoveContainer" containerID="a7efcc09b9005f34d05af2ff13cd52b5d3531141b639d5556415b2a0543ca983" Nov 04 00:59:34.496375 ip-10-0-128-129 kubenswrapper[7059]: E1104 00:59:34.496354 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7efcc09b9005f34d05af2ff13cd52b5d3531141b639d5556415b2a0543ca983\": container with ID starting with a7efcc09b9005f34d05af2ff13cd52b5d3531141b639d5556415b2a0543ca983 not found: ID does not exist" containerID="a7efcc09b9005f34d05af2ff13cd52b5d3531141b639d5556415b2a0543ca983" Nov 04 00:59:34.496475 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.496381 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7efcc09b9005f34d05af2ff13cd52b5d3531141b639d5556415b2a0543ca983"} err="failed to get container status \"a7efcc09b9005f34d05af2ff13cd52b5d3531141b639d5556415b2a0543ca983\": rpc error: code = NotFound desc = could not find container \"a7efcc09b9005f34d05af2ff13cd52b5d3531141b639d5556415b2a0543ca983\": container with ID starting with a7efcc09b9005f34d05af2ff13cd52b5d3531141b639d5556415b2a0543ca983 not found: ID does not exist" Nov 04 00:59:34.508462 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.508437 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69dd78b66f-clp8w"] Nov 04 00:59:34.513577 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.513555 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-69dd78b66f-clp8w"] Nov 04 00:59:34.745757 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:34.745693 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65dcbb7874-lwgl8"] Nov 04 00:59:34.748999 ip-10-0-128-129 kubenswrapper[7059]: W1104 00:59:34.748967 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf880d85_2da0_40fc_a106_22af2ad25cd7.slice/crio-09ac22115fa98b14160d93954a27b48654a7f9d448a31078f1ef7fd291dfa5f0 WatchSource:0}: Error finding container 09ac22115fa98b14160d93954a27b48654a7f9d448a31078f1ef7fd291dfa5f0: Status 404 returned error can't find the container with id 09ac22115fa98b14160d93954a27b48654a7f9d448a31078f1ef7fd291dfa5f0 Nov 04 00:59:35.173754 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:35.173730 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34aecbea-ff97-4f23-8336-326316720a2f" path="/var/lib/kubelet/pods/34aecbea-ff97-4f23-8336-326316720a2f/volumes" Nov 04 00:59:35.482643 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:35.482604 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65dcbb7874-lwgl8" event={"ID":"df880d85-2da0-40fc-a106-22af2ad25cd7","Type":"ContainerStarted","Data":"8fc3c9b3dfb78e0d49a28321350b8c54784e596384e970de088c8b060dba4ee0"} Nov 04 00:59:35.483084 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:35.482663 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65dcbb7874-lwgl8" event={"ID":"df880d85-2da0-40fc-a106-22af2ad25cd7","Type":"ContainerStarted","Data":"09ac22115fa98b14160d93954a27b48654a7f9d448a31078f1ef7fd291dfa5f0"} Nov 04 00:59:35.499850 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:35.499810 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-65dcbb7874-lwgl8" podStartSLOduration=1.499796568 podStartE2EDuration="1.499796568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:59:35.499627698 +0000 UTC m=+60.441476602" watchObservedRunningTime="2026-11-04 00:59:35.499796568 +0000 UTC m=+60.441645527" Nov 04 00:59:36.478713 ip-10-0-128-129 kubenswrapper[7059]: E1104 00:59:36.478677 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 00:59:44.474335 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:44.474291 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65dcbb7874-lwgl8" Nov 04 00:59:44.474792 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:44.474353 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65dcbb7874-lwgl8" Nov 04 00:59:44.479023 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:44.479000 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65dcbb7874-lwgl8" Nov 04 00:59:44.518840 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:44.518820 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65dcbb7874-lwgl8" Nov 04 00:59:44.562026 ip-10-0-128-129 kubenswrapper[7059]: I1104 00:59:44.561999 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fb659d84-hqndc"] Nov 04 01:00:06.478899 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:00:06.478860 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:00:09.581796 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:09.581744 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-fb659d84-hqndc" podUID="e8e30c89-6de5-458e-a105-6e37373450cb" containerName="console" containerID="cri-o://0424946a741e63bd8d67407b2f726ed801db70a16e09e123ca5916ed667bf81c" gracePeriod=15 Nov 04 01:00:09.846143 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:09.846122 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fb659d84-hqndc" Nov 04 01:00:09.914360 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:09.914331 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8e30c89-6de5-458e-a105-6e37373450cb-kube-api-access-6p554\" (UniqueName: \"kubernetes.io/projected/e8e30c89-6de5-458e-a105-6e37373450cb-kube-api-access-6p554\") pod \"e8e30c89-6de5-458e-a105-6e37373450cb\" (UID: \"e8e30c89-6de5-458e-a105-6e37373450cb\") " Nov 04 01:00:09.914496 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:09.914380 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8e30c89-6de5-458e-a105-6e37373450cb-console-config\" (UniqueName: \"kubernetes.io/configmap/e8e30c89-6de5-458e-a105-6e37373450cb-console-config\") pod \"e8e30c89-6de5-458e-a105-6e37373450cb\" (UID: \"e8e30c89-6de5-458e-a105-6e37373450cb\") " Nov 04 01:00:09.914496 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:09.914401 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8e30c89-6de5-458e-a105-6e37373450cb-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e8e30c89-6de5-458e-a105-6e37373450cb-console-oauth-config\") pod \"e8e30c89-6de5-458e-a105-6e37373450cb\" (UID: \"e8e30c89-6de5-458e-a105-6e37373450cb\") " Nov 04 01:00:09.914617 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:09.914514 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8e30c89-6de5-458e-a105-6e37373450cb-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e8e30c89-6de5-458e-a105-6e37373450cb-console-serving-cert\") pod \"e8e30c89-6de5-458e-a105-6e37373450cb\" (UID: \"e8e30c89-6de5-458e-a105-6e37373450cb\") " Nov 04 01:00:09.914617 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:09.914551 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8e30c89-6de5-458e-a105-6e37373450cb-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8e30c89-6de5-458e-a105-6e37373450cb-service-ca\") pod \"e8e30c89-6de5-458e-a105-6e37373450cb\" (UID: \"e8e30c89-6de5-458e-a105-6e37373450cb\") " Nov 04 01:00:09.914617 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:09.914574 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8e30c89-6de5-458e-a105-6e37373450cb-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8e30c89-6de5-458e-a105-6e37373450cb-tmp\") pod \"e8e30c89-6de5-458e-a105-6e37373450cb\" (UID: \"e8e30c89-6de5-458e-a105-6e37373450cb\") " Nov 04 01:00:09.914787 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:09.914627 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8e30c89-6de5-458e-a105-6e37373450cb-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e8e30c89-6de5-458e-a105-6e37373450cb-oauth-serving-cert\") pod \"e8e30c89-6de5-458e-a105-6e37373450cb\" (UID: \"e8e30c89-6de5-458e-a105-6e37373450cb\") " Nov 04 01:00:09.914787 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:09.914657 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8e30c89-6de5-458e-a105-6e37373450cb-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8e30c89-6de5-458e-a105-6e37373450cb-trusted-ca-bundle\") pod \"e8e30c89-6de5-458e-a105-6e37373450cb\" (UID: \"e8e30c89-6de5-458e-a105-6e37373450cb\") " Nov 04 01:00:09.914912 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:09.914879 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8e30c89-6de5-458e-a105-6e37373450cb-console-config" pod "e8e30c89-6de5-458e-a105-6e37373450cb" (UID: "e8e30c89-6de5-458e-a105-6e37373450cb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:00:09.914997 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:09.914959 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8e30c89-6de5-458e-a105-6e37373450cb-service-ca" pod "e8e30c89-6de5-458e-a105-6e37373450cb" (UID: "e8e30c89-6de5-458e-a105-6e37373450cb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:00:09.915122 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:09.915043 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e30c89-6de5-458e-a105-6e37373450cb-tmp" pod "e8e30c89-6de5-458e-a105-6e37373450cb" (UID: "e8e30c89-6de5-458e-a105-6e37373450cb"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:00:09.915270 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:09.915239 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8e30c89-6de5-458e-a105-6e37373450cb-oauth-serving-cert" pod "e8e30c89-6de5-458e-a105-6e37373450cb" (UID: "e8e30c89-6de5-458e-a105-6e37373450cb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:00:09.915516 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:09.915492 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8e30c89-6de5-458e-a105-6e37373450cb-trusted-ca-bundle" pod "e8e30c89-6de5-458e-a105-6e37373450cb" (UID: "e8e30c89-6de5-458e-a105-6e37373450cb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:00:09.916667 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:09.916646 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e30c89-6de5-458e-a105-6e37373450cb-kube-api-access-6p554" pod "e8e30c89-6de5-458e-a105-6e37373450cb" (UID: "e8e30c89-6de5-458e-a105-6e37373450cb"). InnerVolumeSpecName "kube-api-access-6p554". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:00:09.916751 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:09.916696 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8e30c89-6de5-458e-a105-6e37373450cb-console-oauth-config" pod "e8e30c89-6de5-458e-a105-6e37373450cb" (UID: "e8e30c89-6de5-458e-a105-6e37373450cb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:00:09.916843 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:09.916819 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8e30c89-6de5-458e-a105-6e37373450cb-console-serving-cert" pod "e8e30c89-6de5-458e-a105-6e37373450cb" (UID: "e8e30c89-6de5-458e-a105-6e37373450cb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:00:10.015959 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:10.015917 7059 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e8e30c89-6de5-458e-a105-6e37373450cb-console-serving-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:00:10.015959 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:10.015959 7059 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e8e30c89-6de5-458e-a105-6e37373450cb-service-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:00:10.016089 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:10.015968 7059 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8e30c89-6de5-458e-a105-6e37373450cb-tmp\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:00:10.016089 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:10.015978 7059 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e8e30c89-6de5-458e-a105-6e37373450cb-oauth-serving-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:00:10.016089 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:10.015986 7059 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8e30c89-6de5-458e-a105-6e37373450cb-trusted-ca-bundle\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:00:10.016089 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:10.015994 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p554\" (UniqueName: \"kubernetes.io/projected/e8e30c89-6de5-458e-a105-6e37373450cb-kube-api-access-6p554\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:00:10.016089 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:10.016004 7059 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e8e30c89-6de5-458e-a105-6e37373450cb-console-config\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:00:10.016089 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:10.016012 7059 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e8e30c89-6de5-458e-a105-6e37373450cb-console-oauth-config\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:00:10.606470 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:10.606434 7059 generic.go:356] "Generic (PLEG): container finished" podID="e8e30c89-6de5-458e-a105-6e37373450cb" containerID="0424946a741e63bd8d67407b2f726ed801db70a16e09e123ca5916ed667bf81c" exitCode=0 Nov 04 01:00:10.606839 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:10.606479 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fb659d84-hqndc" Nov 04 01:00:10.606839 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:10.606516 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fb659d84-hqndc" event={"ID":"e8e30c89-6de5-458e-a105-6e37373450cb","Type":"ContainerDied","Data":"0424946a741e63bd8d67407b2f726ed801db70a16e09e123ca5916ed667bf81c"} Nov 04 01:00:10.606839 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:10.606566 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fb659d84-hqndc" event={"ID":"e8e30c89-6de5-458e-a105-6e37373450cb","Type":"ContainerDied","Data":"75eff6f81a4aabb14b9835e27f1abdae2b5a2c64679f6f4c95ccb9570c6c6440"} Nov 04 01:00:10.606839 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:10.606586 7059 scope.go:122] "RemoveContainer" containerID="0424946a741e63bd8d67407b2f726ed801db70a16e09e123ca5916ed667bf81c" Nov 04 01:00:10.615018 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:10.614998 7059 scope.go:122] "RemoveContainer" containerID="0424946a741e63bd8d67407b2f726ed801db70a16e09e123ca5916ed667bf81c" Nov 04 01:00:10.615286 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:00:10.615267 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0424946a741e63bd8d67407b2f726ed801db70a16e09e123ca5916ed667bf81c\": container with ID starting with 0424946a741e63bd8d67407b2f726ed801db70a16e09e123ca5916ed667bf81c not found: ID does not exist" containerID="0424946a741e63bd8d67407b2f726ed801db70a16e09e123ca5916ed667bf81c" Nov 04 01:00:10.615385 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:10.615294 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0424946a741e63bd8d67407b2f726ed801db70a16e09e123ca5916ed667bf81c"} err="failed to get container status \"0424946a741e63bd8d67407b2f726ed801db70a16e09e123ca5916ed667bf81c\": rpc error: code = NotFound desc = could not find container \"0424946a741e63bd8d67407b2f726ed801db70a16e09e123ca5916ed667bf81c\": container with ID starting with 0424946a741e63bd8d67407b2f726ed801db70a16e09e123ca5916ed667bf81c not found: ID does not exist" Nov 04 01:00:10.626685 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:10.626662 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fb659d84-hqndc"] Nov 04 01:00:10.632144 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:10.632118 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-fb659d84-hqndc"] Nov 04 01:00:11.174487 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:00:11.174454 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8e30c89-6de5-458e-a105-6e37373450cb" path="/var/lib/kubelet/pods/e8e30c89-6de5-458e-a105-6e37373450cb/volumes" Nov 04 01:00:36.478512 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:00:36.478478 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:01:06.478964 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:01:06.478910 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:01:36.478729 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:01:36.478693 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:02:06.479773 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:02:06.479694 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:02:06.530722 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.530693 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712knr8"] Nov 04 01:02:06.531040 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.531027 7059 cpu_manager.go:414] "containerMap: removing container" podUID="e8e30c89-6de5-458e-a105-6e37373450cb" containerName="console" Nov 04 01:02:06.531087 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.531041 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8e30c89-6de5-458e-a105-6e37373450cb" containerName="console" Nov 04 01:02:06.531120 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.531105 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8e30c89-6de5-458e-a105-6e37373450cb" containerName="console" Nov 04 01:02:06.534233 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.534217 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712knr8" Nov 04 01:02:06.538494 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.538475 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 04 01:02:06.538659 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.538647 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 04 01:02:06.538705 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.538657 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-4wndj\"" Nov 04 01:02:06.542336 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.542317 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712knr8"] Nov 04 01:02:06.584163 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.584138 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b410037a-b339-41ef-9a3c-57566d065fe5-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712knr8\" (UID: \"b410037a-b339-41ef-9a3c-57566d065fe5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712knr8" Nov 04 01:02:06.584271 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.584173 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmgdl\" (UniqueName: \"kubernetes.io/projected/b410037a-b339-41ef-9a3c-57566d065fe5-kube-api-access-vmgdl\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712knr8\" (UID: \"b410037a-b339-41ef-9a3c-57566d065fe5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712knr8" Nov 04 01:02:06.584271 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.584209 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b410037a-b339-41ef-9a3c-57566d065fe5-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712knr8\" (UID: \"b410037a-b339-41ef-9a3c-57566d065fe5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712knr8" Nov 04 01:02:06.684901 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.684871 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b410037a-b339-41ef-9a3c-57566d065fe5-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712knr8\" (UID: \"b410037a-b339-41ef-9a3c-57566d065fe5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712knr8" Nov 04 01:02:06.685039 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.684944 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmgdl\" (UniqueName: \"kubernetes.io/projected/b410037a-b339-41ef-9a3c-57566d065fe5-kube-api-access-vmgdl\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712knr8\" (UID: \"b410037a-b339-41ef-9a3c-57566d065fe5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712knr8" Nov 04 01:02:06.685039 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.684996 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b410037a-b339-41ef-9a3c-57566d065fe5-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712knr8\" (UID: \"b410037a-b339-41ef-9a3c-57566d065fe5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712knr8" Nov 04 01:02:06.685292 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.685271 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b410037a-b339-41ef-9a3c-57566d065fe5-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712knr8\" (UID: \"b410037a-b339-41ef-9a3c-57566d065fe5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712knr8" Nov 04 01:02:06.685369 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.685352 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b410037a-b339-41ef-9a3c-57566d065fe5-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712knr8\" (UID: \"b410037a-b339-41ef-9a3c-57566d065fe5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712knr8" Nov 04 01:02:06.693249 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.693224 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmgdl\" (UniqueName: \"kubernetes.io/projected/b410037a-b339-41ef-9a3c-57566d065fe5-kube-api-access-vmgdl\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712knr8\" (UID: \"b410037a-b339-41ef-9a3c-57566d065fe5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712knr8" Nov 04 01:02:06.806577 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.806524 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77wzbv"] Nov 04 01:02:06.809941 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.809909 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77wzbv" Nov 04 01:02:06.813649 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.813629 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-bzkq2\"" Nov 04 01:02:06.817439 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.817422 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77wzbv"] Nov 04 01:02:06.851638 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.851615 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712knr8" Nov 04 01:02:06.886202 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.886168 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bsbx\" (UniqueName: \"kubernetes.io/projected/6012719b-fd41-4edf-93ee-a3be91fdf8bf-kube-api-access-9bsbx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77wzbv\" (UID: \"6012719b-fd41-4edf-93ee-a3be91fdf8bf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77wzbv" Nov 04 01:02:06.886340 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.886217 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6012719b-fd41-4edf-93ee-a3be91fdf8bf-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77wzbv\" (UID: \"6012719b-fd41-4edf-93ee-a3be91fdf8bf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77wzbv" Nov 04 01:02:06.886340 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.886301 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6012719b-fd41-4edf-93ee-a3be91fdf8bf-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77wzbv\" (UID: \"6012719b-fd41-4edf-93ee-a3be91fdf8bf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77wzbv" Nov 04 01:02:06.900526 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.900499 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswzm7"] Nov 04 01:02:06.904846 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.904825 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswzm7" Nov 04 01:02:06.909429 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.909411 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-kshn9\"" Nov 04 01:02:06.917164 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.914733 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswzm7"] Nov 04 01:02:06.987271 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.987243 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bsbx\" (UniqueName: \"kubernetes.io/projected/6012719b-fd41-4edf-93ee-a3be91fdf8bf-kube-api-access-9bsbx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77wzbv\" (UID: \"6012719b-fd41-4edf-93ee-a3be91fdf8bf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77wzbv" Nov 04 01:02:06.987390 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.987286 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a95926f2-7415-4620-8925-db6a4a4e1651-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswzm7\" (UID: \"a95926f2-7415-4620-8925-db6a4a4e1651\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswzm7" Nov 04 01:02:06.987390 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.987309 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a95926f2-7415-4620-8925-db6a4a4e1651-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswzm7\" (UID: \"a95926f2-7415-4620-8925-db6a4a4e1651\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswzm7" Nov 04 01:02:06.987511 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.987426 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6012719b-fd41-4edf-93ee-a3be91fdf8bf-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77wzbv\" (UID: \"6012719b-fd41-4edf-93ee-a3be91fdf8bf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77wzbv" Nov 04 01:02:06.987511 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.987486 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6012719b-fd41-4edf-93ee-a3be91fdf8bf-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77wzbv\" (UID: \"6012719b-fd41-4edf-93ee-a3be91fdf8bf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77wzbv" Nov 04 01:02:06.987619 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.987550 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcvkn\" (UniqueName: \"kubernetes.io/projected/a95926f2-7415-4620-8925-db6a4a4e1651-kube-api-access-dcvkn\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswzm7\" (UID: \"a95926f2-7415-4620-8925-db6a4a4e1651\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswzm7" Nov 04 01:02:06.987820 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.987804 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6012719b-fd41-4edf-93ee-a3be91fdf8bf-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77wzbv\" (UID: \"6012719b-fd41-4edf-93ee-a3be91fdf8bf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77wzbv" Nov 04 01:02:06.987862 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.987853 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6012719b-fd41-4edf-93ee-a3be91fdf8bf-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77wzbv\" (UID: \"6012719b-fd41-4edf-93ee-a3be91fdf8bf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77wzbv" Nov 04 01:02:06.998364 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:06.998337 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bsbx\" (UniqueName: \"kubernetes.io/projected/6012719b-fd41-4edf-93ee-a3be91fdf8bf-kube-api-access-9bsbx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77wzbv\" (UID: \"6012719b-fd41-4edf-93ee-a3be91fdf8bf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77wzbv" Nov 04 01:02:07.002010 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.001989 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apdhh6"] Nov 04 01:02:07.005407 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.005388 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apdhh6" Nov 04 01:02:07.008678 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.008658 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-bd95b\"" Nov 04 01:02:07.014037 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.014011 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apdhh6"] Nov 04 01:02:07.088895 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.088836 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/413b5e08-7c91-4855-af6e-4c2d0e967b9f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apdhh6\" (UID: \"413b5e08-7c91-4855-af6e-4c2d0e967b9f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apdhh6" Nov 04 01:02:07.088895 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.088863 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/413b5e08-7c91-4855-af6e-4c2d0e967b9f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apdhh6\" (UID: \"413b5e08-7c91-4855-af6e-4c2d0e967b9f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apdhh6" Nov 04 01:02:07.088895 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.088890 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcvkn\" (UniqueName: \"kubernetes.io/projected/a95926f2-7415-4620-8925-db6a4a4e1651-kube-api-access-dcvkn\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswzm7\" (UID: \"a95926f2-7415-4620-8925-db6a4a4e1651\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswzm7" Nov 04 01:02:07.089079 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.089028 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a95926f2-7415-4620-8925-db6a4a4e1651-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswzm7\" (UID: \"a95926f2-7415-4620-8925-db6a4a4e1651\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswzm7" Nov 04 01:02:07.089079 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.089053 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a95926f2-7415-4620-8925-db6a4a4e1651-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswzm7\" (UID: \"a95926f2-7415-4620-8925-db6a4a4e1651\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswzm7" Nov 04 01:02:07.089155 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.089096 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd9tn\" (UniqueName: \"kubernetes.io/projected/413b5e08-7c91-4855-af6e-4c2d0e967b9f-kube-api-access-qd9tn\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apdhh6\" (UID: \"413b5e08-7c91-4855-af6e-4c2d0e967b9f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apdhh6" Nov 04 01:02:07.089370 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.089355 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a95926f2-7415-4620-8925-db6a4a4e1651-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswzm7\" (UID: \"a95926f2-7415-4620-8925-db6a4a4e1651\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswzm7" Nov 04 01:02:07.089405 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.089387 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a95926f2-7415-4620-8925-db6a4a4e1651-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswzm7\" (UID: \"a95926f2-7415-4620-8925-db6a4a4e1651\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswzm7" Nov 04 01:02:07.097157 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.097135 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcvkn\" (UniqueName: \"kubernetes.io/projected/a95926f2-7415-4620-8925-db6a4a4e1651-kube-api-access-dcvkn\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswzm7\" (UID: \"a95926f2-7415-4620-8925-db6a4a4e1651\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswzm7" Nov 04 01:02:07.106475 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.106451 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679lcw"] Nov 04 01:02:07.109938 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.109906 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679lcw" Nov 04 01:02:07.113499 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.113473 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-fwtpq\"" Nov 04 01:02:07.118403 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.118382 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679lcw"] Nov 04 01:02:07.121589 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.121567 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712knr8"] Nov 04 01:02:07.125443 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.125423 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77wzbv" Nov 04 01:02:07.125589 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:02:07.125563 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb410037a_b339_41ef_9a3c_57566d065fe5.slice/crio-60bbbd4a8b10d1b839ef4089651d657866918dca8e0e243d711f467b4af8e9c7 WatchSource:0}: Error finding container 60bbbd4a8b10d1b839ef4089651d657866918dca8e0e243d711f467b4af8e9c7: Status 404 returned error can't find the container with id 60bbbd4a8b10d1b839ef4089651d657866918dca8e0e243d711f467b4af8e9c7 Nov 04 01:02:07.127490 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.127475 7059 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:02:07.190239 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.190215 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/413b5e08-7c91-4855-af6e-4c2d0e967b9f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apdhh6\" (UID: \"413b5e08-7c91-4855-af6e-4c2d0e967b9f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apdhh6" Nov 04 01:02:07.190346 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.190247 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/413b5e08-7c91-4855-af6e-4c2d0e967b9f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apdhh6\" (UID: \"413b5e08-7c91-4855-af6e-4c2d0e967b9f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apdhh6" Nov 04 01:02:07.190346 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.190277 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf296\" (UniqueName: \"kubernetes.io/projected/175e0e78-138d-48c8-b813-3862b8fa7583-kube-api-access-bf296\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679lcw\" (UID: \"175e0e78-138d-48c8-b813-3862b8fa7583\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679lcw" Nov 04 01:02:07.190446 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.190348 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/175e0e78-138d-48c8-b813-3862b8fa7583-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679lcw\" (UID: \"175e0e78-138d-48c8-b813-3862b8fa7583\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679lcw" Nov 04 01:02:07.190503 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.190485 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd9tn\" (UniqueName: \"kubernetes.io/projected/413b5e08-7c91-4855-af6e-4c2d0e967b9f-kube-api-access-qd9tn\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apdhh6\" (UID: \"413b5e08-7c91-4855-af6e-4c2d0e967b9f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apdhh6" Nov 04 01:02:07.190576 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.190560 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/175e0e78-138d-48c8-b813-3862b8fa7583-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679lcw\" (UID: \"175e0e78-138d-48c8-b813-3862b8fa7583\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679lcw" Nov 04 01:02:07.190652 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.190627 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/413b5e08-7c91-4855-af6e-4c2d0e967b9f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apdhh6\" (UID: \"413b5e08-7c91-4855-af6e-4c2d0e967b9f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apdhh6" Nov 04 01:02:07.190711 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.190649 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/413b5e08-7c91-4855-af6e-4c2d0e967b9f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apdhh6\" (UID: \"413b5e08-7c91-4855-af6e-4c2d0e967b9f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apdhh6" Nov 04 01:02:07.201779 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.201759 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd9tn\" (UniqueName: \"kubernetes.io/projected/413b5e08-7c91-4855-af6e-4c2d0e967b9f-kube-api-access-qd9tn\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apdhh6\" (UID: \"413b5e08-7c91-4855-af6e-4c2d0e967b9f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apdhh6" Nov 04 01:02:07.227561 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.227538 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswzm7" Nov 04 01:02:07.291472 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.291446 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/175e0e78-138d-48c8-b813-3862b8fa7583-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679lcw\" (UID: \"175e0e78-138d-48c8-b813-3862b8fa7583\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679lcw" Nov 04 01:02:07.291834 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.291484 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf296\" (UniqueName: \"kubernetes.io/projected/175e0e78-138d-48c8-b813-3862b8fa7583-kube-api-access-bf296\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679lcw\" (UID: \"175e0e78-138d-48c8-b813-3862b8fa7583\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679lcw" Nov 04 01:02:07.291834 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.291524 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/175e0e78-138d-48c8-b813-3862b8fa7583-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679lcw\" (UID: \"175e0e78-138d-48c8-b813-3862b8fa7583\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679lcw" Nov 04 01:02:07.291993 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.291885 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/175e0e78-138d-48c8-b813-3862b8fa7583-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679lcw\" (UID: \"175e0e78-138d-48c8-b813-3862b8fa7583\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679lcw" Nov 04 01:02:07.291993 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.291901 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/175e0e78-138d-48c8-b813-3862b8fa7583-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679lcw\" (UID: \"175e0e78-138d-48c8-b813-3862b8fa7583\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679lcw" Nov 04 01:02:07.300494 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.300467 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf296\" (UniqueName: \"kubernetes.io/projected/175e0e78-138d-48c8-b813-3862b8fa7583-kube-api-access-bf296\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679lcw\" (UID: \"175e0e78-138d-48c8-b813-3862b8fa7583\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679lcw" Nov 04 01:02:07.323125 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.323104 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apdhh6" Nov 04 01:02:07.398401 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.398374 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77wzbv"] Nov 04 01:02:07.400979 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:02:07.400955 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6012719b_fd41_4edf_93ee_a3be91fdf8bf.slice/crio-106d4e5d7366b09ba7171323a25a23ffe94cb8a3f7940044b2563add77fe12b5 WatchSource:0}: Error finding container 106d4e5d7366b09ba7171323a25a23ffe94cb8a3f7940044b2563add77fe12b5: Status 404 returned error can't find the container with id 106d4e5d7366b09ba7171323a25a23ffe94cb8a3f7940044b2563add77fe12b5 Nov 04 01:02:07.427319 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.427297 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679lcw" Nov 04 01:02:07.497715 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.497643 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswzm7"] Nov 04 01:02:07.501652 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:02:07.501628 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda95926f2_7415_4620_8925_db6a4a4e1651.slice/crio-82287394feb3552db1de4238b0145abb276e9a8dcefb87d546b7f403f962d3fd WatchSource:0}: Error finding container 82287394feb3552db1de4238b0145abb276e9a8dcefb87d546b7f403f962d3fd: Status 404 returned error can't find the container with id 82287394feb3552db1de4238b0145abb276e9a8dcefb87d546b7f403f962d3fd Nov 04 01:02:07.593869 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.593841 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apdhh6"] Nov 04 01:02:07.596843 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:02:07.596817 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod413b5e08_7c91_4855_af6e_4c2d0e967b9f.slice/crio-a18a385070d1dd20775cb5d5315c7697b399d9d9757c1c620887ccedc73b84c5 WatchSource:0}: Error finding container a18a385070d1dd20775cb5d5315c7697b399d9d9757c1c620887ccedc73b84c5: Status 404 returned error can't find the container with id a18a385070d1dd20775cb5d5315c7697b399d9d9757c1c620887ccedc73b84c5 Nov 04 01:02:07.702369 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:07.702344 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679lcw"] Nov 04 01:02:07.705187 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:02:07.705161 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175e0e78_138d_48c8_b813_3862b8fa7583.slice/crio-c63f95c9555a521ace3508a8553560ff07c99495f60d1580bcd1a8a506602cbb WatchSource:0}: Error finding container c63f95c9555a521ace3508a8553560ff07c99495f60d1580bcd1a8a506602cbb: Status 404 returned error can't find the container with id c63f95c9555a521ace3508a8553560ff07c99495f60d1580bcd1a8a506602cbb Nov 04 01:02:08.045118 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:08.045034 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswzm7" event={"ID":"a95926f2-7415-4620-8925-db6a4a4e1651","Type":"ContainerStarted","Data":"82287394feb3552db1de4238b0145abb276e9a8dcefb87d546b7f403f962d3fd"} Nov 04 01:02:08.046333 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:08.046304 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712knr8" event={"ID":"b410037a-b339-41ef-9a3c-57566d065fe5","Type":"ContainerStarted","Data":"60bbbd4a8b10d1b839ef4089651d657866918dca8e0e243d711f467b4af8e9c7"} Nov 04 01:02:08.047400 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:08.047378 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679lcw" event={"ID":"175e0e78-138d-48c8-b813-3862b8fa7583","Type":"ContainerStarted","Data":"c63f95c9555a521ace3508a8553560ff07c99495f60d1580bcd1a8a506602cbb"} Nov 04 01:02:08.048622 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:08.048600 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apdhh6" event={"ID":"413b5e08-7c91-4855-af6e-4c2d0e967b9f","Type":"ContainerStarted","Data":"a18a385070d1dd20775cb5d5315c7697b399d9d9757c1c620887ccedc73b84c5"} Nov 04 01:02:08.049848 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:08.049823 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77wzbv" event={"ID":"6012719b-fd41-4edf-93ee-a3be91fdf8bf","Type":"ContainerStarted","Data":"106d4e5d7366b09ba7171323a25a23ffe94cb8a3f7940044b2563add77fe12b5"} Nov 04 01:02:09.472153 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.472111 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446svfc"] Nov 04 01:02:09.476405 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.476380 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446svfc" Nov 04 01:02:09.484405 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.484383 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-bhfmq\"" Nov 04 01:02:09.486877 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.486467 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446svfc"] Nov 04 01:02:09.515884 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.515094 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87207087-e3b5-443b-bbe3-1389e918be21-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446svfc\" (UID: \"87207087-e3b5-443b-bbe3-1389e918be21\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446svfc" Nov 04 01:02:09.515884 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.515220 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxxqw\" (UniqueName: \"kubernetes.io/projected/87207087-e3b5-443b-bbe3-1389e918be21-kube-api-access-fxxqw\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446svfc\" (UID: \"87207087-e3b5-443b-bbe3-1389e918be21\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446svfc" Nov 04 01:02:09.515884 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.515333 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87207087-e3b5-443b-bbe3-1389e918be21-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446svfc\" (UID: \"87207087-e3b5-443b-bbe3-1389e918be21\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446svfc" Nov 04 01:02:09.616363 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.616327 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87207087-e3b5-443b-bbe3-1389e918be21-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446svfc\" (UID: \"87207087-e3b5-443b-bbe3-1389e918be21\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446svfc" Nov 04 01:02:09.616603 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.616425 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87207087-e3b5-443b-bbe3-1389e918be21-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446svfc\" (UID: \"87207087-e3b5-443b-bbe3-1389e918be21\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446svfc" Nov 04 01:02:09.616603 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.616462 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxxqw\" (UniqueName: \"kubernetes.io/projected/87207087-e3b5-443b-bbe3-1389e918be21-kube-api-access-fxxqw\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446svfc\" (UID: \"87207087-e3b5-443b-bbe3-1389e918be21\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446svfc" Nov 04 01:02:09.617218 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.617178 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87207087-e3b5-443b-bbe3-1389e918be21-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446svfc\" (UID: \"87207087-e3b5-443b-bbe3-1389e918be21\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446svfc" Nov 04 01:02:09.617470 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.617418 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87207087-e3b5-443b-bbe3-1389e918be21-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446svfc\" (UID: \"87207087-e3b5-443b-bbe3-1389e918be21\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446svfc" Nov 04 01:02:09.627014 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.626987 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxxqw\" (UniqueName: \"kubernetes.io/projected/87207087-e3b5-443b-bbe3-1389e918be21-kube-api-access-fxxqw\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446svfc\" (UID: \"87207087-e3b5-443b-bbe3-1389e918be21\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446svfc" Nov 04 01:02:09.647689 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.646168 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8jg4n"] Nov 04 01:02:09.650652 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.650624 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8jg4n" Nov 04 01:02:09.662498 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.661032 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8jg4n"] Nov 04 01:02:09.684723 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.684693 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-9xfvx\"" Nov 04 01:02:09.717750 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.717495 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcb4p\" (UniqueName: \"kubernetes.io/projected/5dcbf205-4fa7-4dcf-87b1-df4c907d48f3-kube-api-access-tcb4p\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8jg4n\" (UID: \"5dcbf205-4fa7-4dcf-87b1-df4c907d48f3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8jg4n" Nov 04 01:02:09.717750 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.717542 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf205-4fa7-4dcf-87b1-df4c907d48f3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8jg4n\" (UID: \"5dcbf205-4fa7-4dcf-87b1-df4c907d48f3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8jg4n" Nov 04 01:02:09.717750 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.717589 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf205-4fa7-4dcf-87b1-df4c907d48f3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8jg4n\" (UID: \"5dcbf205-4fa7-4dcf-87b1-df4c907d48f3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8jg4n" Nov 04 01:02:09.775475 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.773992 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5"] Nov 04 01:02:09.778360 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.777691 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" Nov 04 01:02:09.788126 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.788099 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-hplhr\"" Nov 04 01:02:09.790675 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.790587 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5"] Nov 04 01:02:09.802193 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.798780 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 04 01:02:09.802193 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.798866 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 04 01:02:09.802193 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.798964 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 04 01:02:09.810855 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.810807 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446svfc" Nov 04 01:02:09.818377 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.818351 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf205-4fa7-4dcf-87b1-df4c907d48f3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8jg4n\" (UID: \"5dcbf205-4fa7-4dcf-87b1-df4c907d48f3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8jg4n" Nov 04 01:02:09.818486 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.818397 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zntw\" (UniqueName: \"kubernetes.io/projected/100334ea-096e-40b5-9804-41e5fbfac86f-kube-api-access-8zntw\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5\" (UID: \"100334ea-096e-40b5-9804-41e5fbfac86f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" Nov 04 01:02:09.818486 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.818448 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf205-4fa7-4dcf-87b1-df4c907d48f3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8jg4n\" (UID: \"5dcbf205-4fa7-4dcf-87b1-df4c907d48f3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8jg4n" Nov 04 01:02:09.818608 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.818482 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/100334ea-096e-40b5-9804-41e5fbfac86f-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5\" (UID: \"100334ea-096e-40b5-9804-41e5fbfac86f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" Nov 04 01:02:09.818608 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.818537 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/100334ea-096e-40b5-9804-41e5fbfac86f-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5\" (UID: \"100334ea-096e-40b5-9804-41e5fbfac86f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" Nov 04 01:02:09.818608 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.818573 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/100334ea-096e-40b5-9804-41e5fbfac86f-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5\" (UID: \"100334ea-096e-40b5-9804-41e5fbfac86f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" Nov 04 01:02:09.818775 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.818622 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/100334ea-096e-40b5-9804-41e5fbfac86f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5\" (UID: \"100334ea-096e-40b5-9804-41e5fbfac86f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" Nov 04 01:02:09.818775 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.818659 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/100334ea-096e-40b5-9804-41e5fbfac86f-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5\" (UID: \"100334ea-096e-40b5-9804-41e5fbfac86f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" Nov 04 01:02:09.818775 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.818693 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/100334ea-096e-40b5-9804-41e5fbfac86f-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5\" (UID: \"100334ea-096e-40b5-9804-41e5fbfac86f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" Nov 04 01:02:09.818775 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.818721 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/100334ea-096e-40b5-9804-41e5fbfac86f-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5\" (UID: \"100334ea-096e-40b5-9804-41e5fbfac86f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" Nov 04 01:02:09.818775 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.818753 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf205-4fa7-4dcf-87b1-df4c907d48f3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8jg4n\" (UID: \"5dcbf205-4fa7-4dcf-87b1-df4c907d48f3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8jg4n" Nov 04 01:02:09.818775 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.818765 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/100334ea-096e-40b5-9804-41e5fbfac86f-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5\" (UID: \"100334ea-096e-40b5-9804-41e5fbfac86f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" Nov 04 01:02:09.819132 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.818884 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcb4p\" (UniqueName: \"kubernetes.io/projected/5dcbf205-4fa7-4dcf-87b1-df4c907d48f3-kube-api-access-tcb4p\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8jg4n\" (UID: \"5dcbf205-4fa7-4dcf-87b1-df4c907d48f3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8jg4n" Nov 04 01:02:09.819132 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.819042 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf205-4fa7-4dcf-87b1-df4c907d48f3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8jg4n\" (UID: \"5dcbf205-4fa7-4dcf-87b1-df4c907d48f3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8jg4n" Nov 04 01:02:09.829768 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.829724 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcb4p\" (UniqueName: \"kubernetes.io/projected/5dcbf205-4fa7-4dcf-87b1-df4c907d48f3-kube-api-access-tcb4p\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8jg4n\" (UID: \"5dcbf205-4fa7-4dcf-87b1-df4c907d48f3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8jg4n" Nov 04 01:02:09.919650 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.919612 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/100334ea-096e-40b5-9804-41e5fbfac86f-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5\" (UID: \"100334ea-096e-40b5-9804-41e5fbfac86f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" Nov 04 01:02:09.919780 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.919690 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/100334ea-096e-40b5-9804-41e5fbfac86f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5\" (UID: \"100334ea-096e-40b5-9804-41e5fbfac86f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" Nov 04 01:02:09.919780 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.919726 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/100334ea-096e-40b5-9804-41e5fbfac86f-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5\" (UID: \"100334ea-096e-40b5-9804-41e5fbfac86f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" Nov 04 01:02:09.919780 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.919762 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/100334ea-096e-40b5-9804-41e5fbfac86f-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5\" (UID: \"100334ea-096e-40b5-9804-41e5fbfac86f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" Nov 04 01:02:09.919970 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.919793 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/100334ea-096e-40b5-9804-41e5fbfac86f-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5\" (UID: \"100334ea-096e-40b5-9804-41e5fbfac86f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" Nov 04 01:02:09.919970 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.919865 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/100334ea-096e-40b5-9804-41e5fbfac86f-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5\" (UID: \"100334ea-096e-40b5-9804-41e5fbfac86f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" Nov 04 01:02:09.919970 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.919905 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zntw\" (UniqueName: \"kubernetes.io/projected/100334ea-096e-40b5-9804-41e5fbfac86f-kube-api-access-8zntw\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5\" (UID: \"100334ea-096e-40b5-9804-41e5fbfac86f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" Nov 04 01:02:09.920127 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.919992 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/100334ea-096e-40b5-9804-41e5fbfac86f-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5\" (UID: \"100334ea-096e-40b5-9804-41e5fbfac86f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" Nov 04 01:02:09.920127 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.920049 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/100334ea-096e-40b5-9804-41e5fbfac86f-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5\" (UID: \"100334ea-096e-40b5-9804-41e5fbfac86f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" Nov 04 01:02:09.920545 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.920522 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/100334ea-096e-40b5-9804-41e5fbfac86f-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5\" (UID: \"100334ea-096e-40b5-9804-41e5fbfac86f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" Nov 04 01:02:09.920777 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.920757 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/100334ea-096e-40b5-9804-41e5fbfac86f-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5\" (UID: \"100334ea-096e-40b5-9804-41e5fbfac86f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" Nov 04 01:02:09.921572 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.921548 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/100334ea-096e-40b5-9804-41e5fbfac86f-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5\" (UID: \"100334ea-096e-40b5-9804-41e5fbfac86f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" Nov 04 01:02:09.921572 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.921562 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/100334ea-096e-40b5-9804-41e5fbfac86f-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5\" (UID: \"100334ea-096e-40b5-9804-41e5fbfac86f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" Nov 04 01:02:09.921715 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.921639 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/100334ea-096e-40b5-9804-41e5fbfac86f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5\" (UID: \"100334ea-096e-40b5-9804-41e5fbfac86f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" Nov 04 01:02:09.927624 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.927600 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/100334ea-096e-40b5-9804-41e5fbfac86f-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5\" (UID: \"100334ea-096e-40b5-9804-41e5fbfac86f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" Nov 04 01:02:09.931086 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.931065 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zntw\" (UniqueName: \"kubernetes.io/projected/100334ea-096e-40b5-9804-41e5fbfac86f-kube-api-access-8zntw\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5\" (UID: \"100334ea-096e-40b5-9804-41e5fbfac86f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" Nov 04 01:02:09.932463 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.932422 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/100334ea-096e-40b5-9804-41e5fbfac86f-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5\" (UID: \"100334ea-096e-40b5-9804-41e5fbfac86f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" Nov 04 01:02:09.942906 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.942856 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/100334ea-096e-40b5-9804-41e5fbfac86f-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5\" (UID: \"100334ea-096e-40b5-9804-41e5fbfac86f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" Nov 04 01:02:09.975177 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:09.974901 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8jg4n" Nov 04 01:02:10.102969 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:10.102937 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" Nov 04 01:02:10.115817 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:10.115790 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446svfc"] Nov 04 01:02:10.134150 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:02:10.134108 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87207087_e3b5_443b_bbe3_1389e918be21.slice/crio-ce8ea6575ac2cb3826af0f533a772c2a5a5598d7cdc1467ebce41350800b5899 WatchSource:0}: Error finding container ce8ea6575ac2cb3826af0f533a772c2a5a5598d7cdc1467ebce41350800b5899: Status 404 returned error can't find the container with id ce8ea6575ac2cb3826af0f533a772c2a5a5598d7cdc1467ebce41350800b5899 Nov 04 01:02:10.282741 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:10.282711 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8jg4n"] Nov 04 01:02:10.288985 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:02:10.288956 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dcbf205_4fa7_4dcf_87b1_df4c907d48f3.slice/crio-6898b01bc9bd7680db1a9aa10c2a98b52466027de4200a9350da060fe6c26bf9 WatchSource:0}: Error finding container 6898b01bc9bd7680db1a9aa10c2a98b52466027de4200a9350da060fe6c26bf9: Status 404 returned error can't find the container with id 6898b01bc9bd7680db1a9aa10c2a98b52466027de4200a9350da060fe6c26bf9 Nov 04 01:02:10.456632 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:10.456601 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5"] Nov 04 01:02:10.460383 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:02:10.460285 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100334ea_096e_40b5_9804_41e5fbfac86f.slice/crio-cc5d022dbb1734d9437abb27824b2e1197a44d3aa2dc9026b0c34807021491a8 WatchSource:0}: Error finding container cc5d022dbb1734d9437abb27824b2e1197a44d3aa2dc9026b0c34807021491a8: Status 404 returned error can't find the container with id cc5d022dbb1734d9437abb27824b2e1197a44d3aa2dc9026b0c34807021491a8 Nov 04 01:02:11.086703 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:11.086660 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" event={"ID":"100334ea-096e-40b5-9804-41e5fbfac86f","Type":"ContainerStarted","Data":"cc5d022dbb1734d9437abb27824b2e1197a44d3aa2dc9026b0c34807021491a8"} Nov 04 01:02:11.088492 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:11.088461 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446svfc" event={"ID":"87207087-e3b5-443b-bbe3-1389e918be21","Type":"ContainerStarted","Data":"ce8ea6575ac2cb3826af0f533a772c2a5a5598d7cdc1467ebce41350800b5899"} Nov 04 01:02:11.090332 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:11.090308 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8jg4n" event={"ID":"5dcbf205-4fa7-4dcf-87b1-df4c907d48f3","Type":"ContainerStarted","Data":"6898b01bc9bd7680db1a9aa10c2a98b52466027de4200a9350da060fe6c26bf9"} Nov 04 01:02:18.220215 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:18.219449 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:02:18.220215 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:18.219541 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:02:18.220215 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:18.219595 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:02:19.176736 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:19.176699 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" event={"ID":"100334ea-096e-40b5-9804-41e5fbfac86f","Type":"ContainerStarted","Data":"832a6a1176cf43f9d324cde45609c66aba8665e8a37fa9dac6d726f94d0263de"} Nov 04 01:02:19.198354 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:19.197948 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" podStartSLOduration=2.444462338 podStartE2EDuration="10.197909241s" totalImagesPullingTime="7.753446903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:02:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:02:19.196117807 +0000 UTC m=+224.137966708" watchObservedRunningTime="2026-11-04 01:02:19.197909241 +0000 UTC m=+224.139758147" Nov 04 01:02:20.103459 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:20.103428 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" Nov 04 01:02:20.109788 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:20.109516 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" Nov 04 01:02:20.187224 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:20.187194 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" Nov 04 01:02:20.188476 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:20.188447 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5" Nov 04 01:02:22.219190 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:22.219121 7059 generic.go:356] "Generic (PLEG): container finished" podID="6012719b-fd41-4edf-93ee-a3be91fdf8bf" containerID="15c3ffc7365ad031cbb7ee1ea7852124752776794a736500e07d359f287a52e8" exitCode=0 Nov 04 01:02:22.219532 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:22.219257 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77wzbv" event={"ID":"6012719b-fd41-4edf-93ee-a3be91fdf8bf","Type":"ContainerDied","Data":"15c3ffc7365ad031cbb7ee1ea7852124752776794a736500e07d359f287a52e8"} Nov 04 01:02:23.228048 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:23.226682 7059 generic.go:356] "Generic (PLEG): container finished" podID="b410037a-b339-41ef-9a3c-57566d065fe5" containerID="d7546c407df0f109cb33af097291a1c7c64aa453cc2ade518e98374c6a29f0ec" exitCode=0 Nov 04 01:02:23.228048 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:23.226778 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712knr8" event={"ID":"b410037a-b339-41ef-9a3c-57566d065fe5","Type":"ContainerDied","Data":"d7546c407df0f109cb33af097291a1c7c64aa453cc2ade518e98374c6a29f0ec"} Nov 04 01:02:24.232696 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:24.232633 7059 generic.go:356] "Generic (PLEG): container finished" podID="6012719b-fd41-4edf-93ee-a3be91fdf8bf" containerID="e9a4d2eb51cd7ec5139687aabff6f3e1dcddc542df6afd365d825b510394592b" exitCode=0 Nov 04 01:02:24.232696 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:24.232676 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77wzbv" event={"ID":"6012719b-fd41-4edf-93ee-a3be91fdf8bf","Type":"ContainerDied","Data":"e9a4d2eb51cd7ec5139687aabff6f3e1dcddc542df6afd365d825b510394592b"} Nov 04 01:02:24.234378 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:24.234354 7059 generic.go:356] "Generic (PLEG): container finished" podID="a95926f2-7415-4620-8925-db6a4a4e1651" containerID="b1e183acfa65c74898923bf5a62efa2da506b1658232a6b5dfcd3255eb7405c1" exitCode=0 Nov 04 01:02:24.234493 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:24.234442 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswzm7" event={"ID":"a95926f2-7415-4620-8925-db6a4a4e1651","Type":"ContainerDied","Data":"b1e183acfa65c74898923bf5a62efa2da506b1658232a6b5dfcd3255eb7405c1"} Nov 04 01:02:24.236151 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:24.236131 7059 generic.go:356] "Generic (PLEG): container finished" podID="175e0e78-138d-48c8-b813-3862b8fa7583" containerID="b96cc2e6b474ebc81a50ad801004e53babe8624f151bbe081abbbdb78f0f768c" exitCode=0 Nov 04 01:02:24.236236 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:24.236193 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679lcw" event={"ID":"175e0e78-138d-48c8-b813-3862b8fa7583","Type":"ContainerDied","Data":"b96cc2e6b474ebc81a50ad801004e53babe8624f151bbe081abbbdb78f0f768c"} Nov 04 01:02:24.237665 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:24.237647 7059 generic.go:356] "Generic (PLEG): container finished" podID="413b5e08-7c91-4855-af6e-4c2d0e967b9f" containerID="aa2ad63b209670ec1dfe62c5d4631f9453f07591cfe4cad36e6ee76af0de479e" exitCode=0 Nov 04 01:02:24.237748 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:24.237671 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apdhh6" event={"ID":"413b5e08-7c91-4855-af6e-4c2d0e967b9f","Type":"ContainerDied","Data":"aa2ad63b209670ec1dfe62c5d4631f9453f07591cfe4cad36e6ee76af0de479e"} Nov 04 01:02:25.244335 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:25.244304 7059 generic.go:356] "Generic (PLEG): container finished" podID="b410037a-b339-41ef-9a3c-57566d065fe5" containerID="79218fdfa6338edbf9023d90c35a2d181eca5a138a2d57ab494538e931f2a8bf" exitCode=0 Nov 04 01:02:25.244723 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:25.244420 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712knr8" event={"ID":"b410037a-b339-41ef-9a3c-57566d065fe5","Type":"ContainerDied","Data":"79218fdfa6338edbf9023d90c35a2d181eca5a138a2d57ab494538e931f2a8bf"} Nov 04 01:02:25.246087 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:25.246066 7059 generic.go:356] "Generic (PLEG): container finished" podID="87207087-e3b5-443b-bbe3-1389e918be21" containerID="9f85f48fa77899b3160dbc1d3b3f5d8dc3b6f119b6d15da97e8e279dc6a07b14" exitCode=0 Nov 04 01:02:25.246199 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:25.246133 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446svfc" event={"ID":"87207087-e3b5-443b-bbe3-1389e918be21","Type":"ContainerDied","Data":"9f85f48fa77899b3160dbc1d3b3f5d8dc3b6f119b6d15da97e8e279dc6a07b14"} Nov 04 01:02:25.248459 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:25.248432 7059 generic.go:356] "Generic (PLEG): container finished" podID="5dcbf205-4fa7-4dcf-87b1-df4c907d48f3" containerID="c407219cf99fbc3a850351bdf2dfbfb74c2994e86812b2ed13488e9301657afa" exitCode=0 Nov 04 01:02:25.248655 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:25.248503 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8jg4n" event={"ID":"5dcbf205-4fa7-4dcf-87b1-df4c907d48f3","Type":"ContainerDied","Data":"c407219cf99fbc3a850351bdf2dfbfb74c2994e86812b2ed13488e9301657afa"} Nov 04 01:02:26.257394 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:26.257357 7059 generic.go:356] "Generic (PLEG): container finished" podID="a95926f2-7415-4620-8925-db6a4a4e1651" containerID="ef3b42769ea617773611444640981c9736357c001ac30c732c8a63210e247429" exitCode=0 Nov 04 01:02:26.257724 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:26.257492 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswzm7" event={"ID":"a95926f2-7415-4620-8925-db6a4a4e1651","Type":"ContainerDied","Data":"ef3b42769ea617773611444640981c9736357c001ac30c732c8a63210e247429"} Nov 04 01:02:26.261954 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:26.261221 7059 generic.go:356] "Generic (PLEG): container finished" podID="175e0e78-138d-48c8-b813-3862b8fa7583" containerID="ff2e6821b02820b953df2b4cda8bc489914ec5c02a4d4d9b4237b70dba38aa4e" exitCode=0 Nov 04 01:02:26.261954 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:26.261305 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679lcw" event={"ID":"175e0e78-138d-48c8-b813-3862b8fa7583","Type":"ContainerDied","Data":"ff2e6821b02820b953df2b4cda8bc489914ec5c02a4d4d9b4237b70dba38aa4e"} Nov 04 01:02:26.265285 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:26.265232 7059 generic.go:356] "Generic (PLEG): container finished" podID="413b5e08-7c91-4855-af6e-4c2d0e967b9f" containerID="b1996de18b56b67772c0f08ceaf7ee1bf70cdc302f983b7ac8e610e9f1da032e" exitCode=0 Nov 04 01:02:26.265380 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:26.265288 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apdhh6" event={"ID":"413b5e08-7c91-4855-af6e-4c2d0e967b9f","Type":"ContainerDied","Data":"b1996de18b56b67772c0f08ceaf7ee1bf70cdc302f983b7ac8e610e9f1da032e"} Nov 04 01:02:30.305001 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:30.304964 7059 generic.go:356] "Generic (PLEG): container finished" podID="87207087-e3b5-443b-bbe3-1389e918be21" containerID="b352f97c30ccd6abc683108e5a7898abd4dd7086fc8550b45fb1cdfa25c60c37" exitCode=0 Nov 04 01:02:30.305493 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:30.305061 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446svfc" event={"ID":"87207087-e3b5-443b-bbe3-1389e918be21","Type":"ContainerDied","Data":"b352f97c30ccd6abc683108e5a7898abd4dd7086fc8550b45fb1cdfa25c60c37"} Nov 04 01:02:31.325710 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:31.325674 7059 generic.go:356] "Generic (PLEG): container finished" podID="5dcbf205-4fa7-4dcf-87b1-df4c907d48f3" containerID="8cb14d6e17d2020b8369870cd5679f65a5714e871eb3d4c78439707a48db965f" exitCode=0 Nov 04 01:02:31.327192 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:31.327158 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8jg4n" event={"ID":"5dcbf205-4fa7-4dcf-87b1-df4c907d48f3","Type":"ContainerDied","Data":"8cb14d6e17d2020b8369870cd5679f65a5714e871eb3d4c78439707a48db965f"} Nov 04 01:02:36.505573 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:02:36.505533 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:02:38.402701 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:38.402098 7059 generic.go:356] "Generic (PLEG): container finished" podID="6012719b-fd41-4edf-93ee-a3be91fdf8bf" containerID="d93a2d4f7638663ba881df705b397212fa3f7cecdedc8da45b3b19e0705227ce" exitCode=0 Nov 04 01:02:38.402701 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:38.402198 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77wzbv" event={"ID":"6012719b-fd41-4edf-93ee-a3be91fdf8bf","Type":"ContainerDied","Data":"d93a2d4f7638663ba881df705b397212fa3f7cecdedc8da45b3b19e0705227ce"} Nov 04 01:02:39.624000 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:39.623653 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77wzbv" Nov 04 01:02:39.733051 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:39.733026 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6012719b-fd41-4edf-93ee-a3be91fdf8bf-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6012719b-fd41-4edf-93ee-a3be91fdf8bf-bundle\") pod \"6012719b-fd41-4edf-93ee-a3be91fdf8bf\" (UID: \"6012719b-fd41-4edf-93ee-a3be91fdf8bf\") " Nov 04 01:02:39.733186 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:39.733086 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6012719b-fd41-4edf-93ee-a3be91fdf8bf-kube-api-access-9bsbx\" (UniqueName: \"kubernetes.io/projected/6012719b-fd41-4edf-93ee-a3be91fdf8bf-kube-api-access-9bsbx\") pod \"6012719b-fd41-4edf-93ee-a3be91fdf8bf\" (UID: \"6012719b-fd41-4edf-93ee-a3be91fdf8bf\") " Nov 04 01:02:39.733186 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:39.733133 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6012719b-fd41-4edf-93ee-a3be91fdf8bf-util\" (UniqueName: \"kubernetes.io/empty-dir/6012719b-fd41-4edf-93ee-a3be91fdf8bf-util\") pod \"6012719b-fd41-4edf-93ee-a3be91fdf8bf\" (UID: \"6012719b-fd41-4edf-93ee-a3be91fdf8bf\") " Nov 04 01:02:39.734013 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:39.733979 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6012719b-fd41-4edf-93ee-a3be91fdf8bf-bundle" pod "6012719b-fd41-4edf-93ee-a3be91fdf8bf" (UID: "6012719b-fd41-4edf-93ee-a3be91fdf8bf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:02:39.737676 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:39.737647 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6012719b-fd41-4edf-93ee-a3be91fdf8bf-kube-api-access-9bsbx" pod "6012719b-fd41-4edf-93ee-a3be91fdf8bf" (UID: "6012719b-fd41-4edf-93ee-a3be91fdf8bf"). InnerVolumeSpecName "kube-api-access-9bsbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:02:39.739068 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:39.738994 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6012719b-fd41-4edf-93ee-a3be91fdf8bf-util" pod "6012719b-fd41-4edf-93ee-a3be91fdf8bf" (UID: "6012719b-fd41-4edf-93ee-a3be91fdf8bf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:02:39.834617 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:39.834544 7059 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6012719b-fd41-4edf-93ee-a3be91fdf8bf-bundle\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:02:39.834617 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:39.834578 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bsbx\" (UniqueName: \"kubernetes.io/projected/6012719b-fd41-4edf-93ee-a3be91fdf8bf-kube-api-access-9bsbx\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:02:39.834617 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:39.834593 7059 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6012719b-fd41-4edf-93ee-a3be91fdf8bf-util\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:02:40.423730 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:40.423028 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77wzbv" Nov 04 01:02:40.423730 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:40.423081 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77wzbv" event={"ID":"6012719b-fd41-4edf-93ee-a3be91fdf8bf","Type":"ContainerDied","Data":"106d4e5d7366b09ba7171323a25a23ffe94cb8a3f7940044b2563add77fe12b5"} Nov 04 01:02:40.423730 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:40.423152 7059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="106d4e5d7366b09ba7171323a25a23ffe94cb8a3f7940044b2563add77fe12b5" Nov 04 01:02:40.431623 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:40.431199 7059 generic.go:356] "Generic (PLEG): container finished" podID="87207087-e3b5-443b-bbe3-1389e918be21" containerID="9989304d7cdbcc96714a96aae236df216783bac8e0221c95c7036e14c26ac44b" exitCode=0 Nov 04 01:02:40.431623 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:40.431293 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446svfc" event={"ID":"87207087-e3b5-443b-bbe3-1389e918be21","Type":"ContainerDied","Data":"9989304d7cdbcc96714a96aae236df216783bac8e0221c95c7036e14c26ac44b"} Nov 04 01:02:40.434253 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:40.434228 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8jg4n" event={"ID":"5dcbf205-4fa7-4dcf-87b1-df4c907d48f3","Type":"ContainerStarted","Data":"77cb24c44e4cdb097c7081bfe0e07be485da04f7ad2d9efe3a1099f7bae4e189"} Nov 04 01:02:40.437488 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:40.437462 7059 generic.go:356] "Generic (PLEG): container finished" podID="b410037a-b339-41ef-9a3c-57566d065fe5" containerID="e7e2d8b7f0d0e7317a1936f1c78c8944c45295837e40487bd1e3212d8e3f6902" exitCode=0 Nov 04 01:02:40.437592 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:40.437492 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712knr8" event={"ID":"b410037a-b339-41ef-9a3c-57566d065fe5","Type":"ContainerDied","Data":"e7e2d8b7f0d0e7317a1936f1c78c8944c45295837e40487bd1e3212d8e3f6902"} Nov 04 01:02:40.482490 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:40.482444 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8jg4n" podStartSLOduration=3.063828726 podStartE2EDuration="31.482430504s" totalImagesPullingTime="28.418601778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:02:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:02:40.481005304 +0000 UTC m=+245.422854207" watchObservedRunningTime="2026-11-04 01:02:40.482430504 +0000 UTC m=+245.424279406" Nov 04 01:02:41.443581 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:41.443552 7059 generic.go:356] "Generic (PLEG): container finished" podID="a95926f2-7415-4620-8925-db6a4a4e1651" containerID="4a67aea1259e2bc2979e6d75c56b6558aa3f5263ff39042e015d7a4b48ea8468" exitCode=0 Nov 04 01:02:41.444046 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:41.443639 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswzm7" event={"ID":"a95926f2-7415-4620-8925-db6a4a4e1651","Type":"ContainerDied","Data":"4a67aea1259e2bc2979e6d75c56b6558aa3f5263ff39042e015d7a4b48ea8468"} Nov 04 01:02:41.445914 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:41.445881 7059 generic.go:356] "Generic (PLEG): container finished" podID="175e0e78-138d-48c8-b813-3862b8fa7583" containerID="28ec776e78ce43aabd194587a45706cfb6ac130fe52d4b5d8a6f0b6c46c9bdb4" exitCode=0 Nov 04 01:02:41.446056 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:41.445975 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679lcw" event={"ID":"175e0e78-138d-48c8-b813-3862b8fa7583","Type":"ContainerDied","Data":"28ec776e78ce43aabd194587a45706cfb6ac130fe52d4b5d8a6f0b6c46c9bdb4"} Nov 04 01:02:41.448153 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:41.448132 7059 generic.go:356] "Generic (PLEG): container finished" podID="413b5e08-7c91-4855-af6e-4c2d0e967b9f" containerID="2e729b8c65d373665335064af8fe05aaafbc37f770a6af40c2b2adf5b3bb054c" exitCode=0 Nov 04 01:02:41.448248 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:41.448212 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apdhh6" event={"ID":"413b5e08-7c91-4855-af6e-4c2d0e967b9f","Type":"ContainerDied","Data":"2e729b8c65d373665335064af8fe05aaafbc37f770a6af40c2b2adf5b3bb054c"} Nov 04 01:02:41.450354 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:41.450333 7059 generic.go:356] "Generic (PLEG): container finished" podID="5dcbf205-4fa7-4dcf-87b1-df4c907d48f3" containerID="77cb24c44e4cdb097c7081bfe0e07be485da04f7ad2d9efe3a1099f7bae4e189" exitCode=0 Nov 04 01:02:41.450441 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:41.450402 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8jg4n" event={"ID":"5dcbf205-4fa7-4dcf-87b1-df4c907d48f3","Type":"ContainerDied","Data":"77cb24c44e4cdb097c7081bfe0e07be485da04f7ad2d9efe3a1099f7bae4e189"} Nov 04 01:02:41.592180 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:41.592154 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712knr8" Nov 04 01:02:41.642271 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:41.642246 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446svfc" Nov 04 01:02:41.751157 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:41.751092 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b410037a-b339-41ef-9a3c-57566d065fe5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b410037a-b339-41ef-9a3c-57566d065fe5-bundle\") pod \"b410037a-b339-41ef-9a3c-57566d065fe5\" (UID: \"b410037a-b339-41ef-9a3c-57566d065fe5\") " Nov 04 01:02:41.751157 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:41.751130 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b410037a-b339-41ef-9a3c-57566d065fe5-kube-api-access-vmgdl\" (UniqueName: \"kubernetes.io/projected/b410037a-b339-41ef-9a3c-57566d065fe5-kube-api-access-vmgdl\") pod \"b410037a-b339-41ef-9a3c-57566d065fe5\" (UID: \"b410037a-b339-41ef-9a3c-57566d065fe5\") " Nov 04 01:02:41.751157 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:41.751150 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87207087-e3b5-443b-bbe3-1389e918be21-util\" (UniqueName: \"kubernetes.io/empty-dir/87207087-e3b5-443b-bbe3-1389e918be21-util\") pod \"87207087-e3b5-443b-bbe3-1389e918be21\" (UID: \"87207087-e3b5-443b-bbe3-1389e918be21\") " Nov 04 01:02:41.751420 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:41.751205 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87207087-e3b5-443b-bbe3-1389e918be21-kube-api-access-fxxqw\" (UniqueName: \"kubernetes.io/projected/87207087-e3b5-443b-bbe3-1389e918be21-kube-api-access-fxxqw\") pod \"87207087-e3b5-443b-bbe3-1389e918be21\" (UID: \"87207087-e3b5-443b-bbe3-1389e918be21\") " Nov 04 01:02:41.751420 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:41.751279 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87207087-e3b5-443b-bbe3-1389e918be21-bundle\" (UniqueName: \"kubernetes.io/empty-dir/87207087-e3b5-443b-bbe3-1389e918be21-bundle\") pod \"87207087-e3b5-443b-bbe3-1389e918be21\" (UID: \"87207087-e3b5-443b-bbe3-1389e918be21\") " Nov 04 01:02:41.751420 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:41.751306 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b410037a-b339-41ef-9a3c-57566d065fe5-util\" (UniqueName: \"kubernetes.io/empty-dir/b410037a-b339-41ef-9a3c-57566d065fe5-util\") pod \"b410037a-b339-41ef-9a3c-57566d065fe5\" (UID: \"b410037a-b339-41ef-9a3c-57566d065fe5\") " Nov 04 01:02:41.751746 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:41.751712 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87207087-e3b5-443b-bbe3-1389e918be21-bundle" pod "87207087-e3b5-443b-bbe3-1389e918be21" (UID: "87207087-e3b5-443b-bbe3-1389e918be21"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:02:41.751879 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:41.751839 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b410037a-b339-41ef-9a3c-57566d065fe5-bundle" pod "b410037a-b339-41ef-9a3c-57566d065fe5" (UID: "b410037a-b339-41ef-9a3c-57566d065fe5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:02:41.753971 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:41.753947 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b410037a-b339-41ef-9a3c-57566d065fe5-kube-api-access-vmgdl" pod "b410037a-b339-41ef-9a3c-57566d065fe5" (UID: "b410037a-b339-41ef-9a3c-57566d065fe5"). InnerVolumeSpecName "kube-api-access-vmgdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:02:41.754060 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:41.753946 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87207087-e3b5-443b-bbe3-1389e918be21-kube-api-access-fxxqw" pod "87207087-e3b5-443b-bbe3-1389e918be21" (UID: "87207087-e3b5-443b-bbe3-1389e918be21"). InnerVolumeSpecName "kube-api-access-fxxqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:02:41.755878 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:41.755852 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87207087-e3b5-443b-bbe3-1389e918be21-util" pod "87207087-e3b5-443b-bbe3-1389e918be21" (UID: "87207087-e3b5-443b-bbe3-1389e918be21"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:02:41.756984 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:41.756959 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b410037a-b339-41ef-9a3c-57566d065fe5-util" pod "b410037a-b339-41ef-9a3c-57566d065fe5" (UID: "b410037a-b339-41ef-9a3c-57566d065fe5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:02:41.852860 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:41.852834 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxxqw\" (UniqueName: \"kubernetes.io/projected/87207087-e3b5-443b-bbe3-1389e918be21-kube-api-access-fxxqw\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:02:41.852860 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:41.852859 7059 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87207087-e3b5-443b-bbe3-1389e918be21-bundle\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:02:41.853007 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:41.852869 7059 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b410037a-b339-41ef-9a3c-57566d065fe5-util\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:02:41.853007 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:41.852876 7059 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b410037a-b339-41ef-9a3c-57566d065fe5-bundle\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:02:41.853007 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:41.852884 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmgdl\" (UniqueName: \"kubernetes.io/projected/b410037a-b339-41ef-9a3c-57566d065fe5-kube-api-access-vmgdl\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:02:41.853007 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:41.852892 7059 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87207087-e3b5-443b-bbe3-1389e918be21-util\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:02:42.456399 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.456370 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712knr8" Nov 04 01:02:42.456399 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.456388 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712knr8" event={"ID":"b410037a-b339-41ef-9a3c-57566d065fe5","Type":"ContainerDied","Data":"60bbbd4a8b10d1b839ef4089651d657866918dca8e0e243d711f467b4af8e9c7"} Nov 04 01:02:42.456842 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.456443 7059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60bbbd4a8b10d1b839ef4089651d657866918dca8e0e243d711f467b4af8e9c7" Nov 04 01:02:42.458297 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.458264 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446svfc" event={"ID":"87207087-e3b5-443b-bbe3-1389e918be21","Type":"ContainerDied","Data":"ce8ea6575ac2cb3826af0f533a772c2a5a5598d7cdc1467ebce41350800b5899"} Nov 04 01:02:42.458416 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.458301 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446svfc" Nov 04 01:02:42.458416 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.458311 7059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce8ea6575ac2cb3826af0f533a772c2a5a5598d7cdc1467ebce41350800b5899" Nov 04 01:02:42.590744 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.590720 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8jg4n" Nov 04 01:02:42.649584 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.649563 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswzm7" Nov 04 01:02:42.652906 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.652881 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apdhh6" Nov 04 01:02:42.666224 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.666203 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679lcw" Nov 04 01:02:42.759019 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.758940 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a95926f2-7415-4620-8925-db6a4a4e1651-kube-api-access-dcvkn\" (UniqueName: \"kubernetes.io/projected/a95926f2-7415-4620-8925-db6a4a4e1651-kube-api-access-dcvkn\") pod \"a95926f2-7415-4620-8925-db6a4a4e1651\" (UID: \"a95926f2-7415-4620-8925-db6a4a4e1651\") " Nov 04 01:02:42.759019 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.758993 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/413b5e08-7c91-4855-af6e-4c2d0e967b9f-kube-api-access-qd9tn\" (UniqueName: \"kubernetes.io/projected/413b5e08-7c91-4855-af6e-4c2d0e967b9f-kube-api-access-qd9tn\") pod \"413b5e08-7c91-4855-af6e-4c2d0e967b9f\" (UID: \"413b5e08-7c91-4855-af6e-4c2d0e967b9f\") " Nov 04 01:02:42.759235 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.759041 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dcbf205-4fa7-4dcf-87b1-df4c907d48f3-kube-api-access-tcb4p\" (UniqueName: \"kubernetes.io/projected/5dcbf205-4fa7-4dcf-87b1-df4c907d48f3-kube-api-access-tcb4p\") pod \"5dcbf205-4fa7-4dcf-87b1-df4c907d48f3\" (UID: \"5dcbf205-4fa7-4dcf-87b1-df4c907d48f3\") " Nov 04 01:02:42.759235 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.759073 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a95926f2-7415-4620-8925-db6a4a4e1651-util\" (UniqueName: \"kubernetes.io/empty-dir/a95926f2-7415-4620-8925-db6a4a4e1651-util\") pod \"a95926f2-7415-4620-8925-db6a4a4e1651\" (UID: \"a95926f2-7415-4620-8925-db6a4a4e1651\") " Nov 04 01:02:42.759235 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.759140 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/413b5e08-7c91-4855-af6e-4c2d0e967b9f-util\" (UniqueName: \"kubernetes.io/empty-dir/413b5e08-7c91-4855-af6e-4c2d0e967b9f-util\") pod \"413b5e08-7c91-4855-af6e-4c2d0e967b9f\" (UID: \"413b5e08-7c91-4855-af6e-4c2d0e967b9f\") " Nov 04 01:02:42.759235 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.759171 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a95926f2-7415-4620-8925-db6a4a4e1651-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a95926f2-7415-4620-8925-db6a4a4e1651-bundle\") pod \"a95926f2-7415-4620-8925-db6a4a4e1651\" (UID: \"a95926f2-7415-4620-8925-db6a4a4e1651\") " Nov 04 01:02:42.759235 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.759199 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dcbf205-4fa7-4dcf-87b1-df4c907d48f3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf205-4fa7-4dcf-87b1-df4c907d48f3-bundle\") pod \"5dcbf205-4fa7-4dcf-87b1-df4c907d48f3\" (UID: \"5dcbf205-4fa7-4dcf-87b1-df4c907d48f3\") " Nov 04 01:02:42.759235 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.759230 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dcbf205-4fa7-4dcf-87b1-df4c907d48f3-util\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf205-4fa7-4dcf-87b1-df4c907d48f3-util\") pod \"5dcbf205-4fa7-4dcf-87b1-df4c907d48f3\" (UID: \"5dcbf205-4fa7-4dcf-87b1-df4c907d48f3\") " Nov 04 01:02:42.759514 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.759272 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/413b5e08-7c91-4855-af6e-4c2d0e967b9f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/413b5e08-7c91-4855-af6e-4c2d0e967b9f-bundle\") pod \"413b5e08-7c91-4855-af6e-4c2d0e967b9f\" (UID: \"413b5e08-7c91-4855-af6e-4c2d0e967b9f\") " Nov 04 01:02:42.760387 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.760326 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a95926f2-7415-4620-8925-db6a4a4e1651-bundle" pod "a95926f2-7415-4620-8925-db6a4a4e1651" (UID: "a95926f2-7415-4620-8925-db6a4a4e1651"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:02:42.760525 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.760459 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/413b5e08-7c91-4855-af6e-4c2d0e967b9f-bundle" pod "413b5e08-7c91-4855-af6e-4c2d0e967b9f" (UID: "413b5e08-7c91-4855-af6e-4c2d0e967b9f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:02:42.760829 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.760799 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dcbf205-4fa7-4dcf-87b1-df4c907d48f3-bundle" pod "5dcbf205-4fa7-4dcf-87b1-df4c907d48f3" (UID: "5dcbf205-4fa7-4dcf-87b1-df4c907d48f3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:02:42.761453 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.761430 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a95926f2-7415-4620-8925-db6a4a4e1651-kube-api-access-dcvkn" pod "a95926f2-7415-4620-8925-db6a4a4e1651" (UID: "a95926f2-7415-4620-8925-db6a4a4e1651"). InnerVolumeSpecName "kube-api-access-dcvkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:02:42.761853 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.761835 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/413b5e08-7c91-4855-af6e-4c2d0e967b9f-kube-api-access-qd9tn" pod "413b5e08-7c91-4855-af6e-4c2d0e967b9f" (UID: "413b5e08-7c91-4855-af6e-4c2d0e967b9f"). InnerVolumeSpecName "kube-api-access-qd9tn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:02:42.762032 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.761851 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dcbf205-4fa7-4dcf-87b1-df4c907d48f3-kube-api-access-tcb4p" pod "5dcbf205-4fa7-4dcf-87b1-df4c907d48f3" (UID: "5dcbf205-4fa7-4dcf-87b1-df4c907d48f3"). InnerVolumeSpecName "kube-api-access-tcb4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:02:42.766890 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.766869 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dcbf205-4fa7-4dcf-87b1-df4c907d48f3-util" pod "5dcbf205-4fa7-4dcf-87b1-df4c907d48f3" (UID: "5dcbf205-4fa7-4dcf-87b1-df4c907d48f3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:02:42.767954 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.767909 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/413b5e08-7c91-4855-af6e-4c2d0e967b9f-util" pod "413b5e08-7c91-4855-af6e-4c2d0e967b9f" (UID: "413b5e08-7c91-4855-af6e-4c2d0e967b9f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:02:42.768107 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.768088 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a95926f2-7415-4620-8925-db6a4a4e1651-util" pod "a95926f2-7415-4620-8925-db6a4a4e1651" (UID: "a95926f2-7415-4620-8925-db6a4a4e1651"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:02:42.859958 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.859914 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/175e0e78-138d-48c8-b813-3862b8fa7583-kube-api-access-bf296\" (UniqueName: \"kubernetes.io/projected/175e0e78-138d-48c8-b813-3862b8fa7583-kube-api-access-bf296\") pod \"175e0e78-138d-48c8-b813-3862b8fa7583\" (UID: \"175e0e78-138d-48c8-b813-3862b8fa7583\") " Nov 04 01:02:42.860056 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.859976 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/175e0e78-138d-48c8-b813-3862b8fa7583-bundle\" (UniqueName: \"kubernetes.io/empty-dir/175e0e78-138d-48c8-b813-3862b8fa7583-bundle\") pod \"175e0e78-138d-48c8-b813-3862b8fa7583\" (UID: \"175e0e78-138d-48c8-b813-3862b8fa7583\") " Nov 04 01:02:42.860056 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.860006 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/175e0e78-138d-48c8-b813-3862b8fa7583-util\" (UniqueName: \"kubernetes.io/empty-dir/175e0e78-138d-48c8-b813-3862b8fa7583-util\") pod \"175e0e78-138d-48c8-b813-3862b8fa7583\" (UID: \"175e0e78-138d-48c8-b813-3862b8fa7583\") " Nov 04 01:02:42.860246 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.860218 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcvkn\" (UniqueName: \"kubernetes.io/projected/a95926f2-7415-4620-8925-db6a4a4e1651-kube-api-access-dcvkn\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:02:42.860246 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.860246 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd9tn\" (UniqueName: \"kubernetes.io/projected/413b5e08-7c91-4855-af6e-4c2d0e967b9f-kube-api-access-qd9tn\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:02:42.860414 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.860261 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcb4p\" (UniqueName: \"kubernetes.io/projected/5dcbf205-4fa7-4dcf-87b1-df4c907d48f3-kube-api-access-tcb4p\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:02:42.860414 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.860273 7059 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a95926f2-7415-4620-8925-db6a4a4e1651-util\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:02:42.860414 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.860286 7059 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/413b5e08-7c91-4855-af6e-4c2d0e967b9f-util\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:02:42.860414 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.860297 7059 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a95926f2-7415-4620-8925-db6a4a4e1651-bundle\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:02:42.860414 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.860292 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/175e0e78-138d-48c8-b813-3862b8fa7583-bundle" pod "175e0e78-138d-48c8-b813-3862b8fa7583" (UID: "175e0e78-138d-48c8-b813-3862b8fa7583"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:02:42.860414 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.860307 7059 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf205-4fa7-4dcf-87b1-df4c907d48f3-bundle\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:02:42.860414 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.860339 7059 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf205-4fa7-4dcf-87b1-df4c907d48f3-util\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:02:42.860414 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.860349 7059 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/413b5e08-7c91-4855-af6e-4c2d0e967b9f-bundle\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:02:42.861829 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.861811 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/175e0e78-138d-48c8-b813-3862b8fa7583-kube-api-access-bf296" pod "175e0e78-138d-48c8-b813-3862b8fa7583" (UID: "175e0e78-138d-48c8-b813-3862b8fa7583"). InnerVolumeSpecName "kube-api-access-bf296". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:02:42.865752 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.865729 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/175e0e78-138d-48c8-b813-3862b8fa7583-util" pod "175e0e78-138d-48c8-b813-3862b8fa7583" (UID: "175e0e78-138d-48c8-b813-3862b8fa7583"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:02:42.961073 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.961046 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf296\" (UniqueName: \"kubernetes.io/projected/175e0e78-138d-48c8-b813-3862b8fa7583-kube-api-access-bf296\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:02:42.961073 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.961071 7059 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/175e0e78-138d-48c8-b813-3862b8fa7583-bundle\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:02:42.961199 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:42.961084 7059 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/175e0e78-138d-48c8-b813-3862b8fa7583-util\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:02:43.464915 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:43.464846 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswzm7" Nov 04 01:02:43.464915 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:43.464847 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cswzm7" event={"ID":"a95926f2-7415-4620-8925-db6a4a4e1651","Type":"ContainerDied","Data":"82287394feb3552db1de4238b0145abb276e9a8dcefb87d546b7f403f962d3fd"} Nov 04 01:02:43.464915 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:43.464905 7059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82287394feb3552db1de4238b0145abb276e9a8dcefb87d546b7f403f962d3fd" Nov 04 01:02:43.466956 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:43.466911 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679lcw" event={"ID":"175e0e78-138d-48c8-b813-3862b8fa7583","Type":"ContainerDied","Data":"c63f95c9555a521ace3508a8553560ff07c99495f60d1580bcd1a8a506602cbb"} Nov 04 01:02:43.466956 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:43.466942 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c679lcw" Nov 04 01:02:43.467114 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:43.466970 7059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c63f95c9555a521ace3508a8553560ff07c99495f60d1580bcd1a8a506602cbb" Nov 04 01:02:43.468975 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:43.468957 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apdhh6" Nov 04 01:02:43.469082 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:43.468950 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apdhh6" event={"ID":"413b5e08-7c91-4855-af6e-4c2d0e967b9f","Type":"ContainerDied","Data":"a18a385070d1dd20775cb5d5315c7697b399d9d9757c1c620887ccedc73b84c5"} Nov 04 01:02:43.469127 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:43.469083 7059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a18a385070d1dd20775cb5d5315c7697b399d9d9757c1c620887ccedc73b84c5" Nov 04 01:02:43.470868 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:43.470846 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8jg4n" event={"ID":"5dcbf205-4fa7-4dcf-87b1-df4c907d48f3","Type":"ContainerDied","Data":"6898b01bc9bd7680db1a9aa10c2a98b52466027de4200a9350da060fe6c26bf9"} Nov 04 01:02:43.470985 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:43.470875 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8jg4n" Nov 04 01:02:43.470985 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:43.470881 7059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6898b01bc9bd7680db1a9aa10c2a98b52466027de4200a9350da060fe6c26bf9" Nov 04 01:02:46.495824 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.495790 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-dgmxr"] Nov 04 01:02:46.496204 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496092 7059 cpu_manager.go:414] "containerMap: removing container" podUID="87207087-e3b5-443b-bbe3-1389e918be21" containerName="util" Nov 04 01:02:46.496204 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496103 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87207087-e3b5-443b-bbe3-1389e918be21" containerName="util" Nov 04 01:02:46.496204 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496111 7059 cpu_manager.go:414] "containerMap: removing container" podUID="a95926f2-7415-4620-8925-db6a4a4e1651" containerName="pull" Nov 04 01:02:46.496204 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496116 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a95926f2-7415-4620-8925-db6a4a4e1651" containerName="pull" Nov 04 01:02:46.496204 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496127 7059 cpu_manager.go:414] "containerMap: removing container" podUID="a95926f2-7415-4620-8925-db6a4a4e1651" containerName="extract" Nov 04 01:02:46.496204 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496131 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a95926f2-7415-4620-8925-db6a4a4e1651" containerName="extract" Nov 04 01:02:46.496204 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496139 7059 cpu_manager.go:414] "containerMap: removing container" podUID="a95926f2-7415-4620-8925-db6a4a4e1651" containerName="util" Nov 04 01:02:46.496204 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496143 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a95926f2-7415-4620-8925-db6a4a4e1651" containerName="util" Nov 04 01:02:46.496204 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496149 7059 cpu_manager.go:414] "containerMap: removing container" podUID="175e0e78-138d-48c8-b813-3862b8fa7583" containerName="pull" Nov 04 01:02:46.496204 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496154 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="175e0e78-138d-48c8-b813-3862b8fa7583" containerName="pull" Nov 04 01:02:46.496204 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496159 7059 cpu_manager.go:414] "containerMap: removing container" podUID="413b5e08-7c91-4855-af6e-4c2d0e967b9f" containerName="extract" Nov 04 01:02:46.496204 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496163 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="413b5e08-7c91-4855-af6e-4c2d0e967b9f" containerName="extract" Nov 04 01:02:46.496204 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496171 7059 cpu_manager.go:414] "containerMap: removing container" podUID="5dcbf205-4fa7-4dcf-87b1-df4c907d48f3" containerName="pull" Nov 04 01:02:46.496204 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496175 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dcbf205-4fa7-4dcf-87b1-df4c907d48f3" containerName="pull" Nov 04 01:02:46.496204 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496181 7059 cpu_manager.go:414] "containerMap: removing container" podUID="5dcbf205-4fa7-4dcf-87b1-df4c907d48f3" containerName="extract" Nov 04 01:02:46.496204 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496185 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dcbf205-4fa7-4dcf-87b1-df4c907d48f3" containerName="extract" Nov 04 01:02:46.496204 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496193 7059 cpu_manager.go:414] "containerMap: removing container" podUID="87207087-e3b5-443b-bbe3-1389e918be21" containerName="pull" Nov 04 01:02:46.496204 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496199 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87207087-e3b5-443b-bbe3-1389e918be21" containerName="pull" Nov 04 01:02:46.496204 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496205 7059 cpu_manager.go:414] "containerMap: removing container" podUID="175e0e78-138d-48c8-b813-3862b8fa7583" containerName="extract" Nov 04 01:02:46.496204 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496209 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="175e0e78-138d-48c8-b813-3862b8fa7583" containerName="extract" Nov 04 01:02:46.496760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496222 7059 cpu_manager.go:414] "containerMap: removing container" podUID="b410037a-b339-41ef-9a3c-57566d065fe5" containerName="pull" Nov 04 01:02:46.496760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496228 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b410037a-b339-41ef-9a3c-57566d065fe5" containerName="pull" Nov 04 01:02:46.496760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496237 7059 cpu_manager.go:414] "containerMap: removing container" podUID="6012719b-fd41-4edf-93ee-a3be91fdf8bf" containerName="extract" Nov 04 01:02:46.496760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496243 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6012719b-fd41-4edf-93ee-a3be91fdf8bf" containerName="extract" Nov 04 01:02:46.496760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496251 7059 cpu_manager.go:414] "containerMap: removing container" podUID="413b5e08-7c91-4855-af6e-4c2d0e967b9f" containerName="util" Nov 04 01:02:46.496760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496255 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="413b5e08-7c91-4855-af6e-4c2d0e967b9f" containerName="util" Nov 04 01:02:46.496760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496265 7059 cpu_manager.go:414] "containerMap: removing container" podUID="b410037a-b339-41ef-9a3c-57566d065fe5" containerName="extract" Nov 04 01:02:46.496760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496269 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b410037a-b339-41ef-9a3c-57566d065fe5" containerName="extract" Nov 04 01:02:46.496760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496283 7059 cpu_manager.go:414] "containerMap: removing container" podUID="6012719b-fd41-4edf-93ee-a3be91fdf8bf" containerName="util" Nov 04 01:02:46.496760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496287 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6012719b-fd41-4edf-93ee-a3be91fdf8bf" containerName="util" Nov 04 01:02:46.496760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496294 7059 cpu_manager.go:414] "containerMap: removing container" podUID="b410037a-b339-41ef-9a3c-57566d065fe5" containerName="util" Nov 04 01:02:46.496760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496298 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b410037a-b339-41ef-9a3c-57566d065fe5" containerName="util" Nov 04 01:02:46.496760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496303 7059 cpu_manager.go:414] "containerMap: removing container" podUID="5dcbf205-4fa7-4dcf-87b1-df4c907d48f3" containerName="util" Nov 04 01:02:46.496760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496306 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dcbf205-4fa7-4dcf-87b1-df4c907d48f3" containerName="util" Nov 04 01:02:46.496760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496312 7059 cpu_manager.go:414] "containerMap: removing container" podUID="175e0e78-138d-48c8-b813-3862b8fa7583" containerName="util" Nov 04 01:02:46.496760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496316 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="175e0e78-138d-48c8-b813-3862b8fa7583" containerName="util" Nov 04 01:02:46.496760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496322 7059 cpu_manager.go:414] "containerMap: removing container" podUID="6012719b-fd41-4edf-93ee-a3be91fdf8bf" containerName="pull" Nov 04 01:02:46.496760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496326 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6012719b-fd41-4edf-93ee-a3be91fdf8bf" containerName="pull" Nov 04 01:02:46.496760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496332 7059 cpu_manager.go:414] "containerMap: removing container" podUID="413b5e08-7c91-4855-af6e-4c2d0e967b9f" containerName="pull" Nov 04 01:02:46.496760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496336 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="413b5e08-7c91-4855-af6e-4c2d0e967b9f" containerName="pull" Nov 04 01:02:46.496760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496342 7059 cpu_manager.go:414] "containerMap: removing container" podUID="87207087-e3b5-443b-bbe3-1389e918be21" containerName="extract" Nov 04 01:02:46.496760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496345 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87207087-e3b5-443b-bbe3-1389e918be21" containerName="extract" Nov 04 01:02:46.496760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496394 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="a95926f2-7415-4620-8925-db6a4a4e1651" containerName="extract" Nov 04 01:02:46.496760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496401 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dcbf205-4fa7-4dcf-87b1-df4c907d48f3" containerName="extract" Nov 04 01:02:46.496760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496407 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="175e0e78-138d-48c8-b813-3862b8fa7583" containerName="extract" Nov 04 01:02:46.496760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496412 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="6012719b-fd41-4edf-93ee-a3be91fdf8bf" containerName="extract" Nov 04 01:02:46.496760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496420 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b410037a-b339-41ef-9a3c-57566d065fe5" containerName="extract" Nov 04 01:02:46.496760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496426 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="87207087-e3b5-443b-bbe3-1389e918be21" containerName="extract" Nov 04 01:02:46.496760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.496431 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="413b5e08-7c91-4855-af6e-4c2d0e967b9f" containerName="extract" Nov 04 01:02:46.499247 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.499228 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dgmxr" Nov 04 01:02:46.507981 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.507956 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-dgmxr"] Nov 04 01:02:46.508536 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.508516 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 04 01:02:46.508536 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.508527 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 04 01:02:46.508655 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.508527 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-64d8s\"" Nov 04 01:02:46.587049 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.587014 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx4n5\" (UniqueName: \"kubernetes.io/projected/db25d5e4-2c15-426f-80dd-f0ecb20cb1bc-kube-api-access-xx4n5\") pod \"jobset-operator-674cd59dc5-dgmxr\" (UID: \"db25d5e4-2c15-426f-80dd-f0ecb20cb1bc\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dgmxr" Nov 04 01:02:46.587184 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.587079 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db25d5e4-2c15-426f-80dd-f0ecb20cb1bc-tmp\") pod \"jobset-operator-674cd59dc5-dgmxr\" (UID: \"db25d5e4-2c15-426f-80dd-f0ecb20cb1bc\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dgmxr" Nov 04 01:02:46.688417 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.688391 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db25d5e4-2c15-426f-80dd-f0ecb20cb1bc-tmp\") pod \"jobset-operator-674cd59dc5-dgmxr\" (UID: \"db25d5e4-2c15-426f-80dd-f0ecb20cb1bc\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dgmxr" Nov 04 01:02:46.688541 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.688464 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx4n5\" (UniqueName: \"kubernetes.io/projected/db25d5e4-2c15-426f-80dd-f0ecb20cb1bc-kube-api-access-xx4n5\") pod \"jobset-operator-674cd59dc5-dgmxr\" (UID: \"db25d5e4-2c15-426f-80dd-f0ecb20cb1bc\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dgmxr" Nov 04 01:02:46.688747 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.688727 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db25d5e4-2c15-426f-80dd-f0ecb20cb1bc-tmp\") pod \"jobset-operator-674cd59dc5-dgmxr\" (UID: \"db25d5e4-2c15-426f-80dd-f0ecb20cb1bc\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dgmxr" Nov 04 01:02:46.696728 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.696697 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx4n5\" (UniqueName: \"kubernetes.io/projected/db25d5e4-2c15-426f-80dd-f0ecb20cb1bc-kube-api-access-xx4n5\") pod \"jobset-operator-674cd59dc5-dgmxr\" (UID: \"db25d5e4-2c15-426f-80dd-f0ecb20cb1bc\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dgmxr" Nov 04 01:02:46.829911 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:46.829830 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dgmxr" Nov 04 01:02:47.101356 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:47.101333 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-dgmxr"] Nov 04 01:02:47.103312 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:02:47.103288 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb25d5e4_2c15_426f_80dd_f0ecb20cb1bc.slice/crio-fa056e0e5ab012f25b76e40f751e94e381d6e3c95d4517137d3f7fa7970837ef WatchSource:0}: Error finding container fa056e0e5ab012f25b76e40f751e94e381d6e3c95d4517137d3f7fa7970837ef: Status 404 returned error can't find the container with id fa056e0e5ab012f25b76e40f751e94e381d6e3c95d4517137d3f7fa7970837ef Nov 04 01:02:47.488472 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:47.488435 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dgmxr" event={"ID":"db25d5e4-2c15-426f-80dd-f0ecb20cb1bc","Type":"ContainerStarted","Data":"fa056e0e5ab012f25b76e40f751e94e381d6e3c95d4517137d3f7fa7970837ef"} Nov 04 01:02:50.503814 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:50.503777 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dgmxr" event={"ID":"db25d5e4-2c15-426f-80dd-f0ecb20cb1bc","Type":"ContainerStarted","Data":"04f77dd81ef0cec18ce8355cb51cfb313ade8368016bc9535a48d58e297455c7"} Nov 04 01:02:50.523666 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:50.523575 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dgmxr" podStartSLOduration=1.355792112 podStartE2EDuration="4.523561486s" totalImagesPullingTime="3.167769374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:02:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:02:50.523434221 +0000 UTC m=+255.465283124" watchObservedRunningTime="2026-11-04 01:02:50.523561486 +0000 UTC m=+255.465410389" Nov 04 01:02:58.042291 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:58.042255 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m"] Nov 04 01:02:58.045807 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:58.045790 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m" Nov 04 01:02:58.055702 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:58.055676 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 04 01:02:58.055812 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:58.055708 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 04 01:02:58.055812 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:58.055716 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 04 01:02:58.055812 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:58.055742 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 04 01:02:58.056040 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:58.056021 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-9rtlr\"" Nov 04 01:02:58.057648 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:58.057631 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 04 01:02:58.079182 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:58.079161 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m"] Nov 04 01:02:58.088270 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:58.088245 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/730d376a-4902-4852-b314-68767aad19cb-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m\" (UID: \"730d376a-4902-4852-b314-68767aad19cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m" Nov 04 01:02:58.088389 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:58.088306 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/730d376a-4902-4852-b314-68767aad19cb-manifests\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m\" (UID: \"730d376a-4902-4852-b314-68767aad19cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m" Nov 04 01:02:58.088389 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:58.088371 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5z59\" (UniqueName: \"kubernetes.io/projected/730d376a-4902-4852-b314-68767aad19cb-kube-api-access-g5z59\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m\" (UID: \"730d376a-4902-4852-b314-68767aad19cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m" Nov 04 01:02:58.088477 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:58.088400 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/730d376a-4902-4852-b314-68767aad19cb-tmp\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m\" (UID: \"730d376a-4902-4852-b314-68767aad19cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m" Nov 04 01:02:58.088477 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:58.088421 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/730d376a-4902-4852-b314-68767aad19cb-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m\" (UID: \"730d376a-4902-4852-b314-68767aad19cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m" Nov 04 01:02:58.088550 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:58.088476 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/730d376a-4902-4852-b314-68767aad19cb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m\" (UID: \"730d376a-4902-4852-b314-68767aad19cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m" Nov 04 01:02:58.189772 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:58.189742 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/730d376a-4902-4852-b314-68767aad19cb-manifests\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m\" (UID: \"730d376a-4902-4852-b314-68767aad19cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m" Nov 04 01:02:58.189914 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:58.189777 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5z59\" (UniqueName: \"kubernetes.io/projected/730d376a-4902-4852-b314-68767aad19cb-kube-api-access-g5z59\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m\" (UID: \"730d376a-4902-4852-b314-68767aad19cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m" Nov 04 01:02:58.189914 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:58.189804 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/730d376a-4902-4852-b314-68767aad19cb-tmp\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m\" (UID: \"730d376a-4902-4852-b314-68767aad19cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m" Nov 04 01:02:58.189914 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:58.189886 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/730d376a-4902-4852-b314-68767aad19cb-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m\" (UID: \"730d376a-4902-4852-b314-68767aad19cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m" Nov 04 01:02:58.190118 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:58.189936 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/730d376a-4902-4852-b314-68767aad19cb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m\" (UID: \"730d376a-4902-4852-b314-68767aad19cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m" Nov 04 01:02:58.190118 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:58.189980 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/730d376a-4902-4852-b314-68767aad19cb-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m\" (UID: \"730d376a-4902-4852-b314-68767aad19cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m" Nov 04 01:02:58.190215 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:58.190195 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/730d376a-4902-4852-b314-68767aad19cb-manifests\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m\" (UID: \"730d376a-4902-4852-b314-68767aad19cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m" Nov 04 01:02:58.192180 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:58.192153 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/730d376a-4902-4852-b314-68767aad19cb-tmp\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m\" (UID: \"730d376a-4902-4852-b314-68767aad19cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m" Nov 04 01:02:58.192428 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:58.192409 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/730d376a-4902-4852-b314-68767aad19cb-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m\" (UID: \"730d376a-4902-4852-b314-68767aad19cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m" Nov 04 01:02:58.192571 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:58.192550 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/730d376a-4902-4852-b314-68767aad19cb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m\" (UID: \"730d376a-4902-4852-b314-68767aad19cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m" Nov 04 01:02:58.192630 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:58.192571 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/730d376a-4902-4852-b314-68767aad19cb-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m\" (UID: \"730d376a-4902-4852-b314-68767aad19cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m" Nov 04 01:02:58.200042 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:58.200018 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5z59\" (UniqueName: \"kubernetes.io/projected/730d376a-4902-4852-b314-68767aad19cb-kube-api-access-g5z59\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m\" (UID: \"730d376a-4902-4852-b314-68767aad19cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m" Nov 04 01:02:58.361470 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:58.361388 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m" Nov 04 01:02:58.674240 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:58.674218 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m"] Nov 04 01:02:58.676192 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:02:58.676160 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod730d376a_4902_4852_b314_68767aad19cb.slice/crio-27b2b4e0a84c50c501d11f541524aed1e7b166f70647380421f67c938a87b74e WatchSource:0}: Error finding container 27b2b4e0a84c50c501d11f541524aed1e7b166f70647380421f67c938a87b74e: Status 404 returned error can't find the container with id 27b2b4e0a84c50c501d11f541524aed1e7b166f70647380421f67c938a87b74e Nov 04 01:02:59.546560 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:59.546527 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m" event={"ID":"730d376a-4902-4852-b314-68767aad19cb","Type":"ContainerStarted","Data":"27b2b4e0a84c50c501d11f541524aed1e7b166f70647380421f67c938a87b74e"} Nov 04 01:02:59.557288 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:59.557263 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-4r4nr"] Nov 04 01:02:59.561915 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:59.561899 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4r4nr" Nov 04 01:02:59.577581 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:59.577559 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 04 01:02:59.577792 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:59.577768 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 04 01:02:59.579042 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:59.579025 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-fhq9w\"" Nov 04 01:02:59.580350 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:59.580328 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-4r4nr"] Nov 04 01:02:59.602412 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:59.602389 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clqp7\" (UniqueName: \"kubernetes.io/projected/ba1d2c7e-87da-4142-82f3-9a378b73f3b3-kube-api-access-clqp7\") pod \"cert-manager-webhook-5dbbbf75cc-4r4nr\" (UID: \"ba1d2c7e-87da-4142-82f3-9a378b73f3b3\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4r4nr" Nov 04 01:02:59.602498 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:59.602422 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba1d2c7e-87da-4142-82f3-9a378b73f3b3-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-4r4nr\" (UID: \"ba1d2c7e-87da-4142-82f3-9a378b73f3b3\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4r4nr" Nov 04 01:02:59.702954 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:59.702916 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clqp7\" (UniqueName: \"kubernetes.io/projected/ba1d2c7e-87da-4142-82f3-9a378b73f3b3-kube-api-access-clqp7\") pod \"cert-manager-webhook-5dbbbf75cc-4r4nr\" (UID: \"ba1d2c7e-87da-4142-82f3-9a378b73f3b3\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4r4nr" Nov 04 01:02:59.703064 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:59.702966 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba1d2c7e-87da-4142-82f3-9a378b73f3b3-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-4r4nr\" (UID: \"ba1d2c7e-87da-4142-82f3-9a378b73f3b3\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4r4nr" Nov 04 01:02:59.721766 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:59.721739 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba1d2c7e-87da-4142-82f3-9a378b73f3b3-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-4r4nr\" (UID: \"ba1d2c7e-87da-4142-82f3-9a378b73f3b3\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4r4nr" Nov 04 01:02:59.721963 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:59.721940 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clqp7\" (UniqueName: \"kubernetes.io/projected/ba1d2c7e-87da-4142-82f3-9a378b73f3b3-kube-api-access-clqp7\") pod \"cert-manager-webhook-5dbbbf75cc-4r4nr\" (UID: \"ba1d2c7e-87da-4142-82f3-9a378b73f3b3\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4r4nr" Nov 04 01:02:59.872998 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:02:59.872912 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4r4nr" Nov 04 01:03:00.185452 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:00.185421 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-4r4nr"] Nov 04 01:03:00.189133 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:03:00.189103 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba1d2c7e_87da_4142_82f3_9a378b73f3b3.slice/crio-1190712263f02e93674f7b918f7ffbf93b67591c4007e4fdd8ead137fcec0508 WatchSource:0}: Error finding container 1190712263f02e93674f7b918f7ffbf93b67591c4007e4fdd8ead137fcec0508: Status 404 returned error can't find the container with id 1190712263f02e93674f7b918f7ffbf93b67591c4007e4fdd8ead137fcec0508 Nov 04 01:03:00.554529 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:00.554449 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4r4nr" event={"ID":"ba1d2c7e-87da-4142-82f3-9a378b73f3b3","Type":"ContainerStarted","Data":"1190712263f02e93674f7b918f7ffbf93b67591c4007e4fdd8ead137fcec0508"} Nov 04 01:03:03.571384 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:03.571308 7059 generic.go:356] "Generic (PLEG): container finished" podID="730d376a-4902-4852-b314-68767aad19cb" containerID="59a1dafc6f6015e5b7045194eb76dfe86b4ab86a37b83903df0c397f5744b2d2" exitCode=0 Nov 04 01:03:03.571822 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:03.571372 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m" event={"ID":"730d376a-4902-4852-b314-68767aad19cb","Type":"ContainerDied","Data":"59a1dafc6f6015e5b7045194eb76dfe86b4ab86a37b83903df0c397f5744b2d2"} Nov 04 01:03:04.578202 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:04.578150 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4r4nr" event={"ID":"ba1d2c7e-87da-4142-82f3-9a378b73f3b3","Type":"ContainerStarted","Data":"7ac38d5ac580ddae81a6822b2253fbeb1395ace908365a679340cb9f0b236999"} Nov 04 01:03:04.578792 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:04.578211 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4r4nr" Nov 04 01:03:04.580454 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:04.580428 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m" event={"ID":"730d376a-4902-4852-b314-68767aad19cb","Type":"ContainerStarted","Data":"daa9d4a1f4b4f4aa80310986553ea19d0ecdfa858836c53906e064a455ef1051"} Nov 04 01:03:04.580586 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:04.580532 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m" Nov 04 01:03:04.600997 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:04.600957 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4r4nr" podStartSLOduration=2.210604821 podStartE2EDuration="5.600946369s" totalImagesPullingTime="3.390341548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:02:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:04.598257424 +0000 UTC m=+269.540106327" watchObservedRunningTime="2026-11-04 01:03:04.600946369 +0000 UTC m=+269.542795264" Nov 04 01:03:04.636221 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:04.636167 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m" podStartSLOduration=1.67864145 podStartE2EDuration="6.636150349s" totalImagesPullingTime="4.4369236s" totalInitContainerRuntime="520.585299ms" isStatefulPod=false podCreationTimestamp="2026-11-04 01:02:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:04.634689524 +0000 UTC m=+269.576538429" watchObservedRunningTime="2026-11-04 01:03:04.636150349 +0000 UTC m=+269.577999256" Nov 04 01:03:06.478895 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:03:06.478850 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:03:09.095964 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:09.095903 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-b9mcd"] Nov 04 01:03:09.148439 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:09.148410 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-b9mcd"] Nov 04 01:03:09.148586 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:09.148509 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b9mcd" Nov 04 01:03:09.157690 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:09.157668 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 04 01:03:09.157797 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:09.157722 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-tntj4\"" Nov 04 01:03:09.157797 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:09.157732 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 04 01:03:09.293258 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:09.293229 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh27v\" (UniqueName: \"kubernetes.io/projected/eb98db7f-8c90-4e73-ba5d-4d7617d69cd2-kube-api-access-vh27v\") pod \"authorino-operator-7b7bb4bc77-b9mcd\" (UID: \"eb98db7f-8c90-4e73-ba5d-4d7617d69cd2\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-b9mcd" Nov 04 01:03:09.394404 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:09.394343 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh27v\" (UniqueName: \"kubernetes.io/projected/eb98db7f-8c90-4e73-ba5d-4d7617d69cd2-kube-api-access-vh27v\") pod \"authorino-operator-7b7bb4bc77-b9mcd\" (UID: \"eb98db7f-8c90-4e73-ba5d-4d7617d69cd2\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-b9mcd" Nov 04 01:03:09.432730 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:09.432701 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh27v\" (UniqueName: \"kubernetes.io/projected/eb98db7f-8c90-4e73-ba5d-4d7617d69cd2-kube-api-access-vh27v\") pod \"authorino-operator-7b7bb4bc77-b9mcd\" (UID: \"eb98db7f-8c90-4e73-ba5d-4d7617d69cd2\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-b9mcd" Nov 04 01:03:09.471723 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:09.471692 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b9mcd" Nov 04 01:03:09.768698 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:09.768675 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-b9mcd"] Nov 04 01:03:09.771062 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:03:09.771036 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb98db7f_8c90_4e73_ba5d_4d7617d69cd2.slice/crio-3ca4c2924269fd236d75cc92ad58f97ffe3b5ab0ee84d304ded5a5dc32b38397 WatchSource:0}: Error finding container 3ca4c2924269fd236d75cc92ad58f97ffe3b5ab0ee84d304ded5a5dc32b38397: Status 404 returned error can't find the container with id 3ca4c2924269fd236d75cc92ad58f97ffe3b5ab0ee84d304ded5a5dc32b38397 Nov 04 01:03:10.591380 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:10.591353 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4r4nr" Nov 04 01:03:10.613152 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:10.613127 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b9mcd" event={"ID":"eb98db7f-8c90-4e73-ba5d-4d7617d69cd2","Type":"ContainerStarted","Data":"3ca4c2924269fd236d75cc92ad58f97ffe3b5ab0ee84d304ded5a5dc32b38397"} Nov 04 01:03:14.634421 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:14.634381 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b9mcd" event={"ID":"eb98db7f-8c90-4e73-ba5d-4d7617d69cd2","Type":"ContainerStarted","Data":"311b7c4b3ba04a398a74f2084f6d2fd2c1c2e8a9a8def11dd40dd9e075f2d66d"} Nov 04 01:03:14.634824 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:14.634537 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b9mcd" Nov 04 01:03:14.651370 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:14.651300 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b9mcd" podStartSLOduration=1.287807263 podStartE2EDuration="5.651290108s" totalImagesPullingTime="4.363482845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:14.650653505 +0000 UTC m=+279.592502408" watchObservedRunningTime="2026-11-04 01:03:14.651290108 +0000 UTC m=+279.593139012" Nov 04 01:03:15.591677 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:15.591647 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m" Nov 04 01:03:24.853790 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:24.853758 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-9zvzx"] Nov 04 01:03:24.858363 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:24.858344 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9zvzx" Nov 04 01:03:24.865960 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:24.865939 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 04 01:03:24.866735 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:24.866714 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 04 01:03:24.867149 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:24.867131 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8gbdw\"" Nov 04 01:03:24.867777 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:24.867753 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-9zvzx"] Nov 04 01:03:25.027027 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:25.026993 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/80b36f39-30fe-4f0e-bfc4-7945c0f4ac97-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-9zvzx\" (UID: \"80b36f39-30fe-4f0e-bfc4-7945c0f4ac97\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9zvzx" Nov 04 01:03:25.027190 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:25.027139 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/80b36f39-30fe-4f0e-bfc4-7945c0f4ac97-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-9zvzx\" (UID: \"80b36f39-30fe-4f0e-bfc4-7945c0f4ac97\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9zvzx" Nov 04 01:03:25.027190 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:25.027174 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzvwj\" (UniqueName: \"kubernetes.io/projected/80b36f39-30fe-4f0e-bfc4-7945c0f4ac97-kube-api-access-tzvwj\") pod \"kuadrant-console-plugin-66c4fb575d-9zvzx\" (UID: \"80b36f39-30fe-4f0e-bfc4-7945c0f4ac97\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9zvzx" Nov 04 01:03:25.127918 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:25.127839 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/80b36f39-30fe-4f0e-bfc4-7945c0f4ac97-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-9zvzx\" (UID: \"80b36f39-30fe-4f0e-bfc4-7945c0f4ac97\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9zvzx" Nov 04 01:03:25.127918 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:25.127876 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzvwj\" (UniqueName: \"kubernetes.io/projected/80b36f39-30fe-4f0e-bfc4-7945c0f4ac97-kube-api-access-tzvwj\") pod \"kuadrant-console-plugin-66c4fb575d-9zvzx\" (UID: \"80b36f39-30fe-4f0e-bfc4-7945c0f4ac97\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9zvzx" Nov 04 01:03:25.127918 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:25.127899 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/80b36f39-30fe-4f0e-bfc4-7945c0f4ac97-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-9zvzx\" (UID: \"80b36f39-30fe-4f0e-bfc4-7945c0f4ac97\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9zvzx" Nov 04 01:03:25.128646 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:25.128563 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/80b36f39-30fe-4f0e-bfc4-7945c0f4ac97-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-9zvzx\" (UID: \"80b36f39-30fe-4f0e-bfc4-7945c0f4ac97\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9zvzx" Nov 04 01:03:25.130436 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:25.130414 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/80b36f39-30fe-4f0e-bfc4-7945c0f4ac97-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-9zvzx\" (UID: \"80b36f39-30fe-4f0e-bfc4-7945c0f4ac97\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9zvzx" Nov 04 01:03:25.135961 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:25.135943 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzvwj\" (UniqueName: \"kubernetes.io/projected/80b36f39-30fe-4f0e-bfc4-7945c0f4ac97-kube-api-access-tzvwj\") pod \"kuadrant-console-plugin-66c4fb575d-9zvzx\" (UID: \"80b36f39-30fe-4f0e-bfc4-7945c0f4ac97\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9zvzx" Nov 04 01:03:25.171367 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:25.171342 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9zvzx" Nov 04 01:03:25.451814 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:25.451728 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-9zvzx"] Nov 04 01:03:25.454023 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:03:25.453997 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80b36f39_30fe_4f0e_bfc4_7945c0f4ac97.slice/crio-19a0b595526e060465f7cba90f176bcf4bc940c1bfbe62cde5e6ebdc7d15283b WatchSource:0}: Error finding container 19a0b595526e060465f7cba90f176bcf4bc940c1bfbe62cde5e6ebdc7d15283b: Status 404 returned error can't find the container with id 19a0b595526e060465f7cba90f176bcf4bc940c1bfbe62cde5e6ebdc7d15283b Nov 04 01:03:25.642067 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:25.642039 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b9mcd" Nov 04 01:03:25.686357 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:25.686322 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9zvzx" event={"ID":"80b36f39-30fe-4f0e-bfc4-7945c0f4ac97","Type":"ContainerStarted","Data":"19a0b595526e060465f7cba90f176bcf4bc940c1bfbe62cde5e6ebdc7d15283b"} Nov 04 01:03:32.107823 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:32.107791 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-85448bfccc-jwqpc"] Nov 04 01:03:32.111615 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:32.111593 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85448bfccc-jwqpc" Nov 04 01:03:32.119750 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:32.119350 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 04 01:03:32.120829 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:32.120805 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 04 01:03:32.121531 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:32.121508 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-47vck\"" Nov 04 01:03:32.123314 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:32.123293 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85448bfccc-jwqpc"] Nov 04 01:03:32.189539 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:32.189500 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzk4b\" (UniqueName: \"kubernetes.io/projected/104eb9c5-8190-47a2-b0ce-ead26831913c-kube-api-access-dzk4b\") pod \"kube-auth-proxy-85448bfccc-jwqpc\" (UID: \"104eb9c5-8190-47a2-b0ce-ead26831913c\") " pod="openshift-ingress/kube-auth-proxy-85448bfccc-jwqpc" Nov 04 01:03:32.189695 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:32.189550 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/104eb9c5-8190-47a2-b0ce-ead26831913c-tmp\") pod \"kube-auth-proxy-85448bfccc-jwqpc\" (UID: \"104eb9c5-8190-47a2-b0ce-ead26831913c\") " pod="openshift-ingress/kube-auth-proxy-85448bfccc-jwqpc" Nov 04 01:03:32.189695 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:32.189640 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/104eb9c5-8190-47a2-b0ce-ead26831913c-tls-certs\") pod \"kube-auth-proxy-85448bfccc-jwqpc\" (UID: \"104eb9c5-8190-47a2-b0ce-ead26831913c\") " pod="openshift-ingress/kube-auth-proxy-85448bfccc-jwqpc" Nov 04 01:03:32.290592 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:32.290555 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/104eb9c5-8190-47a2-b0ce-ead26831913c-tls-certs\") pod \"kube-auth-proxy-85448bfccc-jwqpc\" (UID: \"104eb9c5-8190-47a2-b0ce-ead26831913c\") " pod="openshift-ingress/kube-auth-proxy-85448bfccc-jwqpc" Nov 04 01:03:32.290763 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:32.290657 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzk4b\" (UniqueName: \"kubernetes.io/projected/104eb9c5-8190-47a2-b0ce-ead26831913c-kube-api-access-dzk4b\") pod \"kube-auth-proxy-85448bfccc-jwqpc\" (UID: \"104eb9c5-8190-47a2-b0ce-ead26831913c\") " pod="openshift-ingress/kube-auth-proxy-85448bfccc-jwqpc" Nov 04 01:03:32.290763 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:32.290691 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/104eb9c5-8190-47a2-b0ce-ead26831913c-tmp\") pod \"kube-auth-proxy-85448bfccc-jwqpc\" (UID: \"104eb9c5-8190-47a2-b0ce-ead26831913c\") " pod="openshift-ingress/kube-auth-proxy-85448bfccc-jwqpc" Nov 04 01:03:32.293083 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:32.293055 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/104eb9c5-8190-47a2-b0ce-ead26831913c-tmp\") pod \"kube-auth-proxy-85448bfccc-jwqpc\" (UID: \"104eb9c5-8190-47a2-b0ce-ead26831913c\") " pod="openshift-ingress/kube-auth-proxy-85448bfccc-jwqpc" Nov 04 01:03:32.293368 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:32.293351 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/104eb9c5-8190-47a2-b0ce-ead26831913c-tls-certs\") pod \"kube-auth-proxy-85448bfccc-jwqpc\" (UID: \"104eb9c5-8190-47a2-b0ce-ead26831913c\") " pod="openshift-ingress/kube-auth-proxy-85448bfccc-jwqpc" Nov 04 01:03:32.303717 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:32.303690 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzk4b\" (UniqueName: \"kubernetes.io/projected/104eb9c5-8190-47a2-b0ce-ead26831913c-kube-api-access-dzk4b\") pod \"kube-auth-proxy-85448bfccc-jwqpc\" (UID: \"104eb9c5-8190-47a2-b0ce-ead26831913c\") " pod="openshift-ingress/kube-auth-proxy-85448bfccc-jwqpc" Nov 04 01:03:32.427837 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:32.427801 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85448bfccc-jwqpc" Nov 04 01:03:32.702445 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:32.702424 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85448bfccc-jwqpc"] Nov 04 01:03:32.704713 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:03:32.704686 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod104eb9c5_8190_47a2_b0ce_ead26831913c.slice/crio-2064fdb7ed013fc87e4a782bb8471be1c1cc5db6712f67de1f0a8de495ab3526 WatchSource:0}: Error finding container 2064fdb7ed013fc87e4a782bb8471be1c1cc5db6712f67de1f0a8de495ab3526: Status 404 returned error can't find the container with id 2064fdb7ed013fc87e4a782bb8471be1c1cc5db6712f67de1f0a8de495ab3526 Nov 04 01:03:32.722845 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:32.722798 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9zvzx" event={"ID":"80b36f39-30fe-4f0e-bfc4-7945c0f4ac97","Type":"ContainerStarted","Data":"fc872b840da16fe3eda198f9928b9231a2a3eb2df0c5ec8dbf75af35e4eb02d4"} Nov 04 01:03:32.724192 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:32.724170 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85448bfccc-jwqpc" event={"ID":"104eb9c5-8190-47a2-b0ce-ead26831913c","Type":"ContainerStarted","Data":"2064fdb7ed013fc87e4a782bb8471be1c1cc5db6712f67de1f0a8de495ab3526"} Nov 04 01:03:32.740016 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:32.739972 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9zvzx" podStartSLOduration=2.324727423 podStartE2EDuration="8.7399569s" totalImagesPullingTime="6.415229477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:32.738942463 +0000 UTC m=+297.680791366" watchObservedRunningTime="2026-11-04 01:03:32.7399569 +0000 UTC m=+297.681805840" Nov 04 01:03:36.175107 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:36.175078 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 04 01:03:36.480002 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:03:36.479964 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:03:36.756300 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:36.756219 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85448bfccc-jwqpc" event={"ID":"104eb9c5-8190-47a2-b0ce-ead26831913c","Type":"ContainerStarted","Data":"7b0cbd0b54e575ad3495a1491a56eafa2072f5b71e5312feb00028e8b72cb434"} Nov 04 01:03:36.774052 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:03:36.774008 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-85448bfccc-jwqpc" podStartSLOduration=1.309682461 podStartE2EDuration="4.773994925s" totalImagesPullingTime="3.464312464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:36.77290221 +0000 UTC m=+301.714751113" watchObservedRunningTime="2026-11-04 01:03:36.773994925 +0000 UTC m=+301.715843828" Nov 04 01:04:06.479515 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:04:06.479474 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:04:28.332780 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.332749 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-4xgpz"] Nov 04 01:04:28.335163 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.335147 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-4xgpz" Nov 04 01:04:28.344219 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.344201 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 04 01:04:28.346887 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.346867 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 04 01:04:28.350093 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.350073 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-z297x\"" Nov 04 01:04:28.350490 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.350468 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-4xgpz"] Nov 04 01:04:28.351085 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.351072 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 04 01:04:28.406968 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.406939 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-wlxsc"] Nov 04 01:04:28.409284 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.409268 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-wlxsc" Nov 04 01:04:28.423876 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.423854 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-wlxsc"] Nov 04 01:04:28.439870 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.439845 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-k6ljb"] Nov 04 01:04:28.442171 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.442153 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-k6ljb" Nov 04 01:04:28.456745 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.456306 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 04 01:04:28.456745 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.456348 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a4d968b-87e6-43b5-b788-5ed8ce9da490-cert\") pod \"kserve-controller-manager-776cc66d47-4xgpz\" (UID: \"0a4d968b-87e6-43b5-b788-5ed8ce9da490\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4xgpz" Nov 04 01:04:28.456745 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.456507 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrrrh\" (UniqueName: \"kubernetes.io/projected/0a4d968b-87e6-43b5-b788-5ed8ce9da490-kube-api-access-rrrrh\") pod \"kserve-controller-manager-776cc66d47-4xgpz\" (UID: \"0a4d968b-87e6-43b5-b788-5ed8ce9da490\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4xgpz" Nov 04 01:04:28.456745 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.456605 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-p2sc2\"" Nov 04 01:04:28.456745 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.456648 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-p49lm\"" Nov 04 01:04:28.456745 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.456717 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 04 01:04:28.457143 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.456790 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 04 01:04:28.460129 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.460106 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-k6ljb"] Nov 04 01:04:28.557874 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.557835 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6dhg\" (UniqueName: \"kubernetes.io/projected/73ab959e-6d9d-47fd-9e3b-aba4bca8451f-kube-api-access-c6dhg\") pod \"model-serving-api-75bd89fdfc-wlxsc\" (UID: \"73ab959e-6d9d-47fd-9e3b-aba4bca8451f\") " pod="opendatahub/model-serving-api-75bd89fdfc-wlxsc" Nov 04 01:04:28.558078 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.557881 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73ab959e-6d9d-47fd-9e3b-aba4bca8451f-tls-certs\") pod \"model-serving-api-75bd89fdfc-wlxsc\" (UID: \"73ab959e-6d9d-47fd-9e3b-aba4bca8451f\") " pod="opendatahub/model-serving-api-75bd89fdfc-wlxsc" Nov 04 01:04:28.558078 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.557976 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98a21413-61fe-4fc7-afc0-1984fd885d6a-cert\") pod \"odh-model-controller-769875db56-k6ljb\" (UID: \"98a21413-61fe-4fc7-afc0-1984fd885d6a\") " pod="opendatahub/odh-model-controller-769875db56-k6ljb" Nov 04 01:04:28.558078 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.558014 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a4d968b-87e6-43b5-b788-5ed8ce9da490-cert\") pod \"kserve-controller-manager-776cc66d47-4xgpz\" (UID: \"0a4d968b-87e6-43b5-b788-5ed8ce9da490\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4xgpz" Nov 04 01:04:28.558247 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.558118 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9r5j\" (UniqueName: \"kubernetes.io/projected/98a21413-61fe-4fc7-afc0-1984fd885d6a-kube-api-access-j9r5j\") pod \"odh-model-controller-769875db56-k6ljb\" (UID: \"98a21413-61fe-4fc7-afc0-1984fd885d6a\") " pod="opendatahub/odh-model-controller-769875db56-k6ljb" Nov 04 01:04:28.558247 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.558151 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrrrh\" (UniqueName: \"kubernetes.io/projected/0a4d968b-87e6-43b5-b788-5ed8ce9da490-kube-api-access-rrrrh\") pod \"kserve-controller-manager-776cc66d47-4xgpz\" (UID: \"0a4d968b-87e6-43b5-b788-5ed8ce9da490\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4xgpz" Nov 04 01:04:28.558247 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.558195 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98a21413-61fe-4fc7-afc0-1984fd885d6a-metrics-certs\") pod \"odh-model-controller-769875db56-k6ljb\" (UID: \"98a21413-61fe-4fc7-afc0-1984fd885d6a\") " pod="opendatahub/odh-model-controller-769875db56-k6ljb" Nov 04 01:04:28.560751 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.560730 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a4d968b-87e6-43b5-b788-5ed8ce9da490-cert\") pod \"kserve-controller-manager-776cc66d47-4xgpz\" (UID: \"0a4d968b-87e6-43b5-b788-5ed8ce9da490\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4xgpz" Nov 04 01:04:28.572974 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.572951 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrrrh\" (UniqueName: \"kubernetes.io/projected/0a4d968b-87e6-43b5-b788-5ed8ce9da490-kube-api-access-rrrrh\") pod \"kserve-controller-manager-776cc66d47-4xgpz\" (UID: \"0a4d968b-87e6-43b5-b788-5ed8ce9da490\") " pod="opendatahub/kserve-controller-manager-776cc66d47-4xgpz" Nov 04 01:04:28.647140 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.647083 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-4xgpz" Nov 04 01:04:28.659435 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.659413 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9r5j\" (UniqueName: \"kubernetes.io/projected/98a21413-61fe-4fc7-afc0-1984fd885d6a-kube-api-access-j9r5j\") pod \"odh-model-controller-769875db56-k6ljb\" (UID: \"98a21413-61fe-4fc7-afc0-1984fd885d6a\") " pod="opendatahub/odh-model-controller-769875db56-k6ljb" Nov 04 01:04:28.659504 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.659456 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98a21413-61fe-4fc7-afc0-1984fd885d6a-metrics-certs\") pod \"odh-model-controller-769875db56-k6ljb\" (UID: \"98a21413-61fe-4fc7-afc0-1984fd885d6a\") " pod="opendatahub/odh-model-controller-769875db56-k6ljb" Nov 04 01:04:28.659540 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.659500 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6dhg\" (UniqueName: \"kubernetes.io/projected/73ab959e-6d9d-47fd-9e3b-aba4bca8451f-kube-api-access-c6dhg\") pod \"model-serving-api-75bd89fdfc-wlxsc\" (UID: \"73ab959e-6d9d-47fd-9e3b-aba4bca8451f\") " pod="opendatahub/model-serving-api-75bd89fdfc-wlxsc" Nov 04 01:04:28.659540 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.659522 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73ab959e-6d9d-47fd-9e3b-aba4bca8451f-tls-certs\") pod \"model-serving-api-75bd89fdfc-wlxsc\" (UID: \"73ab959e-6d9d-47fd-9e3b-aba4bca8451f\") " pod="opendatahub/model-serving-api-75bd89fdfc-wlxsc" Nov 04 01:04:28.659540 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.659537 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98a21413-61fe-4fc7-afc0-1984fd885d6a-cert\") pod \"odh-model-controller-769875db56-k6ljb\" (UID: \"98a21413-61fe-4fc7-afc0-1984fd885d6a\") " pod="opendatahub/odh-model-controller-769875db56-k6ljb" Nov 04 01:04:28.662170 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.662150 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73ab959e-6d9d-47fd-9e3b-aba4bca8451f-tls-certs\") pod \"model-serving-api-75bd89fdfc-wlxsc\" (UID: \"73ab959e-6d9d-47fd-9e3b-aba4bca8451f\") " pod="opendatahub/model-serving-api-75bd89fdfc-wlxsc" Nov 04 01:04:28.662525 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.662505 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98a21413-61fe-4fc7-afc0-1984fd885d6a-metrics-certs\") pod \"odh-model-controller-769875db56-k6ljb\" (UID: \"98a21413-61fe-4fc7-afc0-1984fd885d6a\") " pod="opendatahub/odh-model-controller-769875db56-k6ljb" Nov 04 01:04:28.662611 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.662538 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98a21413-61fe-4fc7-afc0-1984fd885d6a-cert\") pod \"odh-model-controller-769875db56-k6ljb\" (UID: \"98a21413-61fe-4fc7-afc0-1984fd885d6a\") " pod="opendatahub/odh-model-controller-769875db56-k6ljb" Nov 04 01:04:28.671598 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.671577 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9r5j\" (UniqueName: \"kubernetes.io/projected/98a21413-61fe-4fc7-afc0-1984fd885d6a-kube-api-access-j9r5j\") pod \"odh-model-controller-769875db56-k6ljb\" (UID: \"98a21413-61fe-4fc7-afc0-1984fd885d6a\") " pod="opendatahub/odh-model-controller-769875db56-k6ljb" Nov 04 01:04:28.671803 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.671775 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6dhg\" (UniqueName: \"kubernetes.io/projected/73ab959e-6d9d-47fd-9e3b-aba4bca8451f-kube-api-access-c6dhg\") pod \"model-serving-api-75bd89fdfc-wlxsc\" (UID: \"73ab959e-6d9d-47fd-9e3b-aba4bca8451f\") " pod="opendatahub/model-serving-api-75bd89fdfc-wlxsc" Nov 04 01:04:28.720275 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.720239 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-wlxsc" Nov 04 01:04:28.763109 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.763074 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-k6ljb" Nov 04 01:04:28.928626 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:28.928582 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-4xgpz"] Nov 04 01:04:28.942160 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:04:28.942127 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4d968b_87e6_43b5_b788_5ed8ce9da490.slice/crio-5dfd486d796805d4311dd6769983acb185de92a39dd0a5fa2164da146ab4c48f WatchSource:0}: Error finding container 5dfd486d796805d4311dd6769983acb185de92a39dd0a5fa2164da146ab4c48f: Status 404 returned error can't find the container with id 5dfd486d796805d4311dd6769983acb185de92a39dd0a5fa2164da146ab4c48f Nov 04 01:04:29.010001 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:29.009979 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-wlxsc"] Nov 04 01:04:29.012396 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:29.012363 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-4xgpz" event={"ID":"0a4d968b-87e6-43b5-b788-5ed8ce9da490","Type":"ContainerStarted","Data":"5dfd486d796805d4311dd6769983acb185de92a39dd0a5fa2164da146ab4c48f"} Nov 04 01:04:29.012534 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:04:29.012512 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73ab959e_6d9d_47fd_9e3b_aba4bca8451f.slice/crio-f36f32a23ccdc9886c0d407562d7aef2c69e51d35c17af08fc0510c3b2b53e3a WatchSource:0}: Error finding container f36f32a23ccdc9886c0d407562d7aef2c69e51d35c17af08fc0510c3b2b53e3a: Status 404 returned error can't find the container with id f36f32a23ccdc9886c0d407562d7aef2c69e51d35c17af08fc0510c3b2b53e3a Nov 04 01:04:29.061626 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:29.061605 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-k6ljb"] Nov 04 01:04:29.063833 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:04:29.063808 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a21413_61fe_4fc7_afc0_1984fd885d6a.slice/crio-c4e664b1cfc7b900f70fc82df4b5cec028ae46c04e4c89435a17e7f169c7f63d WatchSource:0}: Error finding container c4e664b1cfc7b900f70fc82df4b5cec028ae46c04e4c89435a17e7f169c7f63d: Status 404 returned error can't find the container with id c4e664b1cfc7b900f70fc82df4b5cec028ae46c04e4c89435a17e7f169c7f63d Nov 04 01:04:30.024212 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:30.024166 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-wlxsc" event={"ID":"73ab959e-6d9d-47fd-9e3b-aba4bca8451f","Type":"ContainerStarted","Data":"f36f32a23ccdc9886c0d407562d7aef2c69e51d35c17af08fc0510c3b2b53e3a"} Nov 04 01:04:30.025755 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:30.025722 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-k6ljb" event={"ID":"98a21413-61fe-4fc7-afc0-1984fd885d6a","Type":"ContainerStarted","Data":"c4e664b1cfc7b900f70fc82df4b5cec028ae46c04e4c89435a17e7f169c7f63d"} Nov 04 01:04:32.040641 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:32.040553 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-wlxsc" event={"ID":"73ab959e-6d9d-47fd-9e3b-aba4bca8451f","Type":"ContainerStarted","Data":"4c2d06db831174c1a30855556942232c7fb16f2e32b90881a8cd2cf8bb316d47"} Nov 04 01:04:32.041428 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:32.041405 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-wlxsc" Nov 04 01:04:32.059199 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:32.059145 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-wlxsc" podStartSLOduration=1.370810191 podStartE2EDuration="4.059125643s" totalImagesPullingTime="2.688315452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:04:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:04:32.057335351 +0000 UTC m=+356.999184255" watchObservedRunningTime="2026-11-04 01:04:32.059125643 +0000 UTC m=+357.000974546" Nov 04 01:04:33.048823 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:33.048788 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-4xgpz" event={"ID":"0a4d968b-87e6-43b5-b788-5ed8ce9da490","Type":"ContainerStarted","Data":"7dd93acccd75deeb37c85365dbcf130a2726cefe59509583a9fedba22d187393"} Nov 04 01:04:33.049437 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:33.048859 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-4xgpz" Nov 04 01:04:33.051621 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:33.051560 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-k6ljb" event={"ID":"98a21413-61fe-4fc7-afc0-1984fd885d6a","Type":"ContainerStarted","Data":"492f4534922af056104357f0195977f6df8cd88a75fd6f91cae02e6ff7a9213b"} Nov 04 01:04:33.051977 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:33.051648 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-k6ljb" Nov 04 01:04:33.068632 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:33.068568 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-4xgpz" podStartSLOduration=1.5429287619999998 podStartE2EDuration="5.068544851s" totalImagesPullingTime="3.525616089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:04:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:04:33.066438883 +0000 UTC m=+358.008287786" watchObservedRunningTime="2026-11-04 01:04:33.068544851 +0000 UTC m=+358.010393753" Nov 04 01:04:33.082680 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:33.082589 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-k6ljb" podStartSLOduration=1.320079622 podStartE2EDuration="5.08257584s" totalImagesPullingTime="3.762496218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:04:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:04:33.080980675 +0000 UTC m=+358.022829577" watchObservedRunningTime="2026-11-04 01:04:33.08257584 +0000 UTC m=+358.024424746" Nov 04 01:04:35.208676 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:35.208650 7059 scope.go:122] "RemoveContainer" containerID="5d9a88f9c5dde43cfecfc6ea742cafb158898614f734f5010a0b9464fc63c044" Nov 04 01:04:35.216795 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:35.216777 7059 scope.go:122] "RemoveContainer" containerID="afae6491578532a4bef9c8acd31e9b111e435252399947ae1f2fd00ca59089ca" Nov 04 01:04:35.224179 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:35.224165 7059 scope.go:122] "RemoveContainer" containerID="080c05bf8c8df8fd6944be22f641c4387b191c82c18e2f3ae7ae692294c8c524" Nov 04 01:04:35.231263 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:35.231247 7059 scope.go:122] "RemoveContainer" containerID="ab68ecb00853ae770662fdd183ad9f573a55ee801404bdab3b6a5ece6b989e3d" Nov 04 01:04:35.238776 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:35.238759 7059 scope.go:122] "RemoveContainer" containerID="e906195f126edeb824eec62dc18fff594496021354341d88597607dba034b945" Nov 04 01:04:35.247119 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:35.247101 7059 scope.go:122] "RemoveContainer" containerID="1c1a21c4d2257675634b3e4d9ef1aa95b4b9e7f748cf77988f3085f0a4b9bc14" Nov 04 01:04:35.254321 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:35.254302 7059 scope.go:122] "RemoveContainer" containerID="1449e5c536ddfc17d9303f1dc08eb2d596ecca01af40809b528ae00d0b6f96e1" Nov 04 01:04:36.479994 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:04:36.479956 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:04:44.065877 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:44.065846 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-k6ljb" Nov 04 01:04:44.067783 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:04:44.067763 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-wlxsc" Nov 04 01:05:04.068852 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:05:04.068773 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-4xgpz" Nov 04 01:05:06.479107 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:05:06.479072 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:05:36.479142 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:05:36.479105 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:06:06.479521 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:06:06.479485 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:06:36.479394 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:06:36.479356 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:07:06.478958 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:07:06.478900 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:07:36.479711 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:07:36.479658 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:08:06.479340 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:08:06.479254 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:08:36.479486 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:08:36.479451 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:09:06.479575 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:09:06.479535 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:09:36.478882 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:09:36.478844 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:10:06.479596 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:10:06.479553 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:10:36.479013 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:10:36.478976 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:11:06.479405 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:11:06.479333 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:11:36.479646 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:11:36.479605 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:12:06.479900 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:12:06.479863 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:12:36.479204 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:12:36.479169 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:13:06.479439 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:13:06.479404 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:13:36.480110 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:13:36.480075 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:14:06.478739 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:14:06.478707 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:14:08.653405 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:08.653374 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-c6vhz"] Nov 04 01:14:08.656891 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:08.656870 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-c6vhz" Nov 04 01:14:08.664830 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:08.664802 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 04 01:14:08.666403 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:08.666378 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-c6vhz"] Nov 04 01:14:08.723120 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:08.723085 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f15a64fb-779f-45fd-a2b3-11ea13d7aed0-config-file\") pod \"limitador-limitador-74dfcfb894-c6vhz\" (UID: \"f15a64fb-779f-45fd-a2b3-11ea13d7aed0\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-c6vhz" Nov 04 01:14:08.723302 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:08.723166 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzrnp\" (UniqueName: \"kubernetes.io/projected/f15a64fb-779f-45fd-a2b3-11ea13d7aed0-kube-api-access-mzrnp\") pod \"limitador-limitador-74dfcfb894-c6vhz\" (UID: \"f15a64fb-779f-45fd-a2b3-11ea13d7aed0\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-c6vhz" Nov 04 01:14:08.757648 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:08.757616 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-c6vhz"] Nov 04 01:14:08.824201 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:08.824171 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f15a64fb-779f-45fd-a2b3-11ea13d7aed0-config-file\") pod \"limitador-limitador-74dfcfb894-c6vhz\" (UID: \"f15a64fb-779f-45fd-a2b3-11ea13d7aed0\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-c6vhz" Nov 04 01:14:08.824347 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:08.824221 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzrnp\" (UniqueName: \"kubernetes.io/projected/f15a64fb-779f-45fd-a2b3-11ea13d7aed0-kube-api-access-mzrnp\") pod \"limitador-limitador-74dfcfb894-c6vhz\" (UID: \"f15a64fb-779f-45fd-a2b3-11ea13d7aed0\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-c6vhz" Nov 04 01:14:08.824858 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:08.824839 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f15a64fb-779f-45fd-a2b3-11ea13d7aed0-config-file\") pod \"limitador-limitador-74dfcfb894-c6vhz\" (UID: \"f15a64fb-779f-45fd-a2b3-11ea13d7aed0\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-c6vhz" Nov 04 01:14:08.832508 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:08.832483 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzrnp\" (UniqueName: \"kubernetes.io/projected/f15a64fb-779f-45fd-a2b3-11ea13d7aed0-kube-api-access-mzrnp\") pod \"limitador-limitador-74dfcfb894-c6vhz\" (UID: \"f15a64fb-779f-45fd-a2b3-11ea13d7aed0\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-c6vhz" Nov 04 01:14:08.971706 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:08.971673 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-c6vhz" Nov 04 01:14:09.243097 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:09.243075 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-c6vhz"] Nov 04 01:14:09.245211 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:14:09.245186 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf15a64fb_779f_45fd_a2b3_11ea13d7aed0.slice/crio-8c5ee739e374408979ccaa1cda191b2b0f287da772d057fac5d0d52f3deb0b97 WatchSource:0}: Error finding container 8c5ee739e374408979ccaa1cda191b2b0f287da772d057fac5d0d52f3deb0b97: Status 404 returned error can't find the container with id 8c5ee739e374408979ccaa1cda191b2b0f287da772d057fac5d0d52f3deb0b97 Nov 04 01:14:09.246992 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:09.246978 7059 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:14:09.437165 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:09.437140 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-kzpf8"] Nov 04 01:14:09.441735 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:09.441717 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-kzpf8" Nov 04 01:14:09.446609 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:09.446588 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-kzpf8"] Nov 04 01:14:09.450402 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:09.450378 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2xvwz\"" Nov 04 01:14:09.530183 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:09.530101 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66fsm\" (UniqueName: \"kubernetes.io/projected/dbbfd64f-4c20-4169-b90f-ec0f942ee796-kube-api-access-66fsm\") pod \"authorino-57f7f8659f-kzpf8\" (UID: \"dbbfd64f-4c20-4169-b90f-ec0f942ee796\") " pod="kuadrant-system/authorino-57f7f8659f-kzpf8" Nov 04 01:14:09.631311 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:09.631281 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66fsm\" (UniqueName: \"kubernetes.io/projected/dbbfd64f-4c20-4169-b90f-ec0f942ee796-kube-api-access-66fsm\") pod \"authorino-57f7f8659f-kzpf8\" (UID: \"dbbfd64f-4c20-4169-b90f-ec0f942ee796\") " pod="kuadrant-system/authorino-57f7f8659f-kzpf8" Nov 04 01:14:09.639630 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:09.639605 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66fsm\" (UniqueName: \"kubernetes.io/projected/dbbfd64f-4c20-4169-b90f-ec0f942ee796-kube-api-access-66fsm\") pod \"authorino-57f7f8659f-kzpf8\" (UID: \"dbbfd64f-4c20-4169-b90f-ec0f942ee796\") " pod="kuadrant-system/authorino-57f7f8659f-kzpf8" Nov 04 01:14:09.733657 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:09.733624 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-c6vhz" event={"ID":"f15a64fb-779f-45fd-a2b3-11ea13d7aed0","Type":"ContainerStarted","Data":"8c5ee739e374408979ccaa1cda191b2b0f287da772d057fac5d0d52f3deb0b97"} Nov 04 01:14:09.754819 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:09.754794 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-kzpf8" Nov 04 01:14:10.023088 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:10.023056 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-kzpf8"] Nov 04 01:14:10.025774 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:14:10.025743 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbbfd64f_4c20_4169_b90f_ec0f942ee796.slice/crio-ffbbce1cf0576332a6c4b56942f8392bb818f6a856b20dd075920a8f1973d011 WatchSource:0}: Error finding container ffbbce1cf0576332a6c4b56942f8392bb818f6a856b20dd075920a8f1973d011: Status 404 returned error can't find the container with id ffbbce1cf0576332a6c4b56942f8392bb818f6a856b20dd075920a8f1973d011 Nov 04 01:14:10.741299 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:10.741260 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-kzpf8" event={"ID":"dbbfd64f-4c20-4169-b90f-ec0f942ee796","Type":"ContainerStarted","Data":"ffbbce1cf0576332a6c4b56942f8392bb818f6a856b20dd075920a8f1973d011"} Nov 04 01:14:11.747157 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:11.747129 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-c6vhz" event={"ID":"f15a64fb-779f-45fd-a2b3-11ea13d7aed0","Type":"ContainerStarted","Data":"18d37e60b91934652f6deae2e07f2bcedbe964ceba10ad60118761c889d27da9"} Nov 04 01:14:11.747534 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:11.747215 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-c6vhz" Nov 04 01:14:11.765233 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:11.765187 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-c6vhz" podStartSLOduration=1.937041403 podStartE2EDuration="3.765174853s" totalImagesPullingTime="1.82813345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:14:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:14:11.762897747 +0000 UTC m=+936.704746649" watchObservedRunningTime="2026-11-04 01:14:11.765174853 +0000 UTC m=+936.707023756" Nov 04 01:14:12.753189 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:12.753148 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-kzpf8" event={"ID":"dbbfd64f-4c20-4169-b90f-ec0f942ee796","Type":"ContainerStarted","Data":"fb970163af4c7d25fd1d9946acc0a28e9f71fa7f9d84863116b02789bce4e650"} Nov 04 01:14:12.768320 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:12.768272 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-kzpf8" podStartSLOduration=1.508356831 podStartE2EDuration="3.768257491s" totalImagesPullingTime="2.25990066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:14:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:14:12.767222388 +0000 UTC m=+937.709071292" watchObservedRunningTime="2026-11-04 01:14:12.768257491 +0000 UTC m=+937.710106393" Nov 04 01:14:12.837008 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:12.836975 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-kzpf8"] Nov 04 01:14:14.762988 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:14.762946 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-kzpf8" podUID="dbbfd64f-4c20-4169-b90f-ec0f942ee796" containerName="authorino" containerID="cri-o://fb970163af4c7d25fd1d9946acc0a28e9f71fa7f9d84863116b02789bce4e650" gracePeriod=30 Nov 04 01:14:15.002764 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:15.002739 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-kzpf8" Nov 04 01:14:15.077120 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:15.077054 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbbfd64f-4c20-4169-b90f-ec0f942ee796-kube-api-access-66fsm\" (UniqueName: \"kubernetes.io/projected/dbbfd64f-4c20-4169-b90f-ec0f942ee796-kube-api-access-66fsm\") pod \"dbbfd64f-4c20-4169-b90f-ec0f942ee796\" (UID: \"dbbfd64f-4c20-4169-b90f-ec0f942ee796\") " Nov 04 01:14:15.079319 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:15.079291 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbbfd64f-4c20-4169-b90f-ec0f942ee796-kube-api-access-66fsm" pod "dbbfd64f-4c20-4169-b90f-ec0f942ee796" (UID: "dbbfd64f-4c20-4169-b90f-ec0f942ee796"). InnerVolumeSpecName "kube-api-access-66fsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:14:15.177657 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:15.177634 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66fsm\" (UniqueName: \"kubernetes.io/projected/dbbfd64f-4c20-4169-b90f-ec0f942ee796-kube-api-access-66fsm\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:14:15.771494 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:15.771452 7059 generic.go:356] "Generic (PLEG): container finished" podID="dbbfd64f-4c20-4169-b90f-ec0f942ee796" containerID="fb970163af4c7d25fd1d9946acc0a28e9f71fa7f9d84863116b02789bce4e650" exitCode=0 Nov 04 01:14:15.771907 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:15.771634 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-kzpf8" event={"ID":"dbbfd64f-4c20-4169-b90f-ec0f942ee796","Type":"ContainerDied","Data":"fb970163af4c7d25fd1d9946acc0a28e9f71fa7f9d84863116b02789bce4e650"} Nov 04 01:14:15.771907 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:15.771679 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-kzpf8" event={"ID":"dbbfd64f-4c20-4169-b90f-ec0f942ee796","Type":"ContainerDied","Data":"ffbbce1cf0576332a6c4b56942f8392bb818f6a856b20dd075920a8f1973d011"} Nov 04 01:14:15.771907 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:15.771702 7059 scope.go:122] "RemoveContainer" containerID="fb970163af4c7d25fd1d9946acc0a28e9f71fa7f9d84863116b02789bce4e650" Nov 04 01:14:15.771907 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:15.771880 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-kzpf8" Nov 04 01:14:15.781764 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:15.781744 7059 scope.go:122] "RemoveContainer" containerID="fb970163af4c7d25fd1d9946acc0a28e9f71fa7f9d84863116b02789bce4e650" Nov 04 01:14:15.782051 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:14:15.782030 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb970163af4c7d25fd1d9946acc0a28e9f71fa7f9d84863116b02789bce4e650\": container with ID starting with fb970163af4c7d25fd1d9946acc0a28e9f71fa7f9d84863116b02789bce4e650 not found: ID does not exist" containerID="fb970163af4c7d25fd1d9946acc0a28e9f71fa7f9d84863116b02789bce4e650" Nov 04 01:14:15.782129 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:15.782058 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb970163af4c7d25fd1d9946acc0a28e9f71fa7f9d84863116b02789bce4e650"} err="failed to get container status \"fb970163af4c7d25fd1d9946acc0a28e9f71fa7f9d84863116b02789bce4e650\": rpc error: code = NotFound desc = could not find container \"fb970163af4c7d25fd1d9946acc0a28e9f71fa7f9d84863116b02789bce4e650\": container with ID starting with fb970163af4c7d25fd1d9946acc0a28e9f71fa7f9d84863116b02789bce4e650 not found: ID does not exist" Nov 04 01:14:15.789778 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:15.789756 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-kzpf8"] Nov 04 01:14:15.794774 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:15.794751 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-kzpf8"] Nov 04 01:14:17.176201 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:17.176166 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbbfd64f-4c20-4169-b90f-ec0f942ee796" path="/var/lib/kubelet/pods/dbbfd64f-4c20-4169-b90f-ec0f942ee796/volumes" Nov 04 01:14:22.754616 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:22.754590 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-c6vhz" Nov 04 01:14:24.536838 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:24.536802 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-c6vhz"] Nov 04 01:14:24.537272 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:24.537005 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-c6vhz" podUID="f15a64fb-779f-45fd-a2b3-11ea13d7aed0" containerName="limitador" containerID="cri-o://18d37e60b91934652f6deae2e07f2bcedbe964ceba10ad60118761c889d27da9" gracePeriod=30 Nov 04 01:14:25.083650 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:25.083624 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-c6vhz" Nov 04 01:14:25.261033 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:25.261008 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f15a64fb-779f-45fd-a2b3-11ea13d7aed0-kube-api-access-mzrnp\" (UniqueName: \"kubernetes.io/projected/f15a64fb-779f-45fd-a2b3-11ea13d7aed0-kube-api-access-mzrnp\") pod \"f15a64fb-779f-45fd-a2b3-11ea13d7aed0\" (UID: \"f15a64fb-779f-45fd-a2b3-11ea13d7aed0\") " Nov 04 01:14:25.261183 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:25.261048 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f15a64fb-779f-45fd-a2b3-11ea13d7aed0-config-file\" (UniqueName: \"kubernetes.io/configmap/f15a64fb-779f-45fd-a2b3-11ea13d7aed0-config-file\") pod \"f15a64fb-779f-45fd-a2b3-11ea13d7aed0\" (UID: \"f15a64fb-779f-45fd-a2b3-11ea13d7aed0\") " Nov 04 01:14:25.261441 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:25.261407 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f15a64fb-779f-45fd-a2b3-11ea13d7aed0-config-file" pod "f15a64fb-779f-45fd-a2b3-11ea13d7aed0" (UID: "f15a64fb-779f-45fd-a2b3-11ea13d7aed0"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:14:25.263225 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:25.263203 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f15a64fb-779f-45fd-a2b3-11ea13d7aed0-kube-api-access-mzrnp" pod "f15a64fb-779f-45fd-a2b3-11ea13d7aed0" (UID: "f15a64fb-779f-45fd-a2b3-11ea13d7aed0"). InnerVolumeSpecName "kube-api-access-mzrnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:14:25.361692 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:25.361665 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzrnp\" (UniqueName: \"kubernetes.io/projected/f15a64fb-779f-45fd-a2b3-11ea13d7aed0-kube-api-access-mzrnp\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:14:25.361784 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:25.361693 7059 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f15a64fb-779f-45fd-a2b3-11ea13d7aed0-config-file\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:14:25.822632 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:25.822596 7059 generic.go:356] "Generic (PLEG): container finished" podID="f15a64fb-779f-45fd-a2b3-11ea13d7aed0" containerID="18d37e60b91934652f6deae2e07f2bcedbe964ceba10ad60118761c889d27da9" exitCode=0 Nov 04 01:14:25.823085 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:25.822636 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-c6vhz" event={"ID":"f15a64fb-779f-45fd-a2b3-11ea13d7aed0","Type":"ContainerDied","Data":"18d37e60b91934652f6deae2e07f2bcedbe964ceba10ad60118761c889d27da9"} Nov 04 01:14:25.823085 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:25.822641 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-c6vhz" Nov 04 01:14:25.823085 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:25.822694 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-c6vhz" event={"ID":"f15a64fb-779f-45fd-a2b3-11ea13d7aed0","Type":"ContainerDied","Data":"8c5ee739e374408979ccaa1cda191b2b0f287da772d057fac5d0d52f3deb0b97"} Nov 04 01:14:25.823085 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:25.822717 7059 scope.go:122] "RemoveContainer" containerID="18d37e60b91934652f6deae2e07f2bcedbe964ceba10ad60118761c889d27da9" Nov 04 01:14:25.832188 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:25.832164 7059 scope.go:122] "RemoveContainer" containerID="18d37e60b91934652f6deae2e07f2bcedbe964ceba10ad60118761c889d27da9" Nov 04 01:14:25.832462 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:14:25.832443 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18d37e60b91934652f6deae2e07f2bcedbe964ceba10ad60118761c889d27da9\": container with ID starting with 18d37e60b91934652f6deae2e07f2bcedbe964ceba10ad60118761c889d27da9 not found: ID does not exist" containerID="18d37e60b91934652f6deae2e07f2bcedbe964ceba10ad60118761c889d27da9" Nov 04 01:14:25.832533 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:25.832474 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18d37e60b91934652f6deae2e07f2bcedbe964ceba10ad60118761c889d27da9"} err="failed to get container status \"18d37e60b91934652f6deae2e07f2bcedbe964ceba10ad60118761c889d27da9\": rpc error: code = NotFound desc = could not find container \"18d37e60b91934652f6deae2e07f2bcedbe964ceba10ad60118761c889d27da9\": container with ID starting with 18d37e60b91934652f6deae2e07f2bcedbe964ceba10ad60118761c889d27da9 not found: ID does not exist" Nov 04 01:14:25.844575 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:25.844552 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-c6vhz"] Nov 04 01:14:25.848370 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:25.848353 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-c6vhz"] Nov 04 01:14:27.165129 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:27.165090 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-lchwg"] Nov 04 01:14:27.165496 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:27.165463 7059 cpu_manager.go:414] "containerMap: removing container" podUID="dbbfd64f-4c20-4169-b90f-ec0f942ee796" containerName="authorino" Nov 04 01:14:27.165496 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:27.165476 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbbfd64f-4c20-4169-b90f-ec0f942ee796" containerName="authorino" Nov 04 01:14:27.165496 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:27.165494 7059 cpu_manager.go:414] "containerMap: removing container" podUID="f15a64fb-779f-45fd-a2b3-11ea13d7aed0" containerName="limitador" Nov 04 01:14:27.165597 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:27.165500 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f15a64fb-779f-45fd-a2b3-11ea13d7aed0" containerName="limitador" Nov 04 01:14:27.165597 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:27.165552 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f15a64fb-779f-45fd-a2b3-11ea13d7aed0" containerName="limitador" Nov 04 01:14:27.165597 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:27.165567 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbbfd64f-4c20-4169-b90f-ec0f942ee796" containerName="authorino" Nov 04 01:14:27.170030 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:27.170010 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lchwg" Nov 04 01:14:27.177188 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:27.177158 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f15a64fb-779f-45fd-a2b3-11ea13d7aed0" path="/var/lib/kubelet/pods/f15a64fb-779f-45fd-a2b3-11ea13d7aed0/volumes" Nov 04 01:14:27.178483 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:27.178462 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-lchwg"] Nov 04 01:14:27.178725 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:27.178705 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:14:27.184351 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:27.184331 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:14:27.185424 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:27.185407 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 04 01:14:27.185488 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:27.185475 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-x75w7\"" Nov 04 01:14:27.276683 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:27.276652 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhkh5\" (UniqueName: \"kubernetes.io/projected/31791d64-921a-4126-8c34-6ea469f60e13-kube-api-access-zhkh5\") pod \"postgres-7cdbf6dc44-lchwg\" (UID: \"31791d64-921a-4126-8c34-6ea469f60e13\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lchwg" Nov 04 01:14:27.276834 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:27.276721 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/31791d64-921a-4126-8c34-6ea469f60e13-data\") pod \"postgres-7cdbf6dc44-lchwg\" (UID: \"31791d64-921a-4126-8c34-6ea469f60e13\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lchwg" Nov 04 01:14:27.377509 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:27.377475 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhkh5\" (UniqueName: \"kubernetes.io/projected/31791d64-921a-4126-8c34-6ea469f60e13-kube-api-access-zhkh5\") pod \"postgres-7cdbf6dc44-lchwg\" (UID: \"31791d64-921a-4126-8c34-6ea469f60e13\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lchwg" Nov 04 01:14:27.377672 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:27.377521 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/31791d64-921a-4126-8c34-6ea469f60e13-data\") pod \"postgres-7cdbf6dc44-lchwg\" (UID: \"31791d64-921a-4126-8c34-6ea469f60e13\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lchwg" Nov 04 01:14:27.377900 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:27.377882 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/31791d64-921a-4126-8c34-6ea469f60e13-data\") pod \"postgres-7cdbf6dc44-lchwg\" (UID: \"31791d64-921a-4126-8c34-6ea469f60e13\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lchwg" Nov 04 01:14:27.385434 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:27.385412 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhkh5\" (UniqueName: \"kubernetes.io/projected/31791d64-921a-4126-8c34-6ea469f60e13-kube-api-access-zhkh5\") pod \"postgres-7cdbf6dc44-lchwg\" (UID: \"31791d64-921a-4126-8c34-6ea469f60e13\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lchwg" Nov 04 01:14:27.484085 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:27.484066 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lchwg" Nov 04 01:14:27.756893 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:27.756774 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-lchwg"] Nov 04 01:14:27.759426 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:14:27.759396 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31791d64_921a_4126_8c34_6ea469f60e13.slice/crio-8114d6113c602d3d03ccf59172f1a745be5c03b66882e495c2270dd49427bec9 WatchSource:0}: Error finding container 8114d6113c602d3d03ccf59172f1a745be5c03b66882e495c2270dd49427bec9: Status 404 returned error can't find the container with id 8114d6113c602d3d03ccf59172f1a745be5c03b66882e495c2270dd49427bec9 Nov 04 01:14:27.834097 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:27.834072 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lchwg" event={"ID":"31791d64-921a-4126-8c34-6ea469f60e13","Type":"ContainerStarted","Data":"8114d6113c602d3d03ccf59172f1a745be5c03b66882e495c2270dd49427bec9"} Nov 04 01:14:33.866183 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:33.866152 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lchwg" event={"ID":"31791d64-921a-4126-8c34-6ea469f60e13","Type":"ContainerStarted","Data":"f1ed6e784b5e416741362317a136110669c7ea32a2a13a605bbd7f4c9dd998be"} Nov 04 01:14:33.866724 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:33.866214 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lchwg" Nov 04 01:14:33.883004 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:33.882959 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lchwg" podStartSLOduration=0.987422983 podStartE2EDuration="6.88294571s" totalImagesPullingTime="5.895522727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:14:33.881180055 +0000 UTC m=+958.823028972" watchObservedRunningTime="2026-11-04 01:14:33.88294571 +0000 UTC m=+958.824794604" Nov 04 01:14:36.479582 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:14:36.479542 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:14:39.900303 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:39.900270 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lchwg" Nov 04 01:14:47.044386 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:47.044350 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8b95c9c66-cv97b"] Nov 04 01:14:47.147394 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:47.147360 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8b95c9c66-cv97b"] Nov 04 01:14:47.147541 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:47.147492 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8b95c9c66-cv97b" Nov 04 01:14:47.151335 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:47.151313 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 04 01:14:47.152444 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:47.152425 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 04 01:14:47.152444 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:47.152434 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 04 01:14:47.152582 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:47.152431 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nlglx\"" Nov 04 01:14:47.254833 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:47.254808 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj8v5\" (UniqueName: \"kubernetes.io/projected/288d4d03-2c1e-4e8c-85c8-138613edd7d4-kube-api-access-vj8v5\") pod \"maas-controller-8b95c9c66-cv97b\" (UID: \"288d4d03-2c1e-4e8c-85c8-138613edd7d4\") " pod="opendatahub/maas-controller-8b95c9c66-cv97b" Nov 04 01:14:47.254979 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:47.254865 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/288d4d03-2c1e-4e8c-85c8-138613edd7d4-metrics-certs\") pod \"maas-controller-8b95c9c66-cv97b\" (UID: \"288d4d03-2c1e-4e8c-85c8-138613edd7d4\") " pod="opendatahub/maas-controller-8b95c9c66-cv97b" Nov 04 01:14:47.255072 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:47.255055 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/288d4d03-2c1e-4e8c-85c8-138613edd7d4-webhook-cert\") pod \"maas-controller-8b95c9c66-cv97b\" (UID: \"288d4d03-2c1e-4e8c-85c8-138613edd7d4\") " pod="opendatahub/maas-controller-8b95c9c66-cv97b" Nov 04 01:14:47.356159 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:47.356092 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj8v5\" (UniqueName: \"kubernetes.io/projected/288d4d03-2c1e-4e8c-85c8-138613edd7d4-kube-api-access-vj8v5\") pod \"maas-controller-8b95c9c66-cv97b\" (UID: \"288d4d03-2c1e-4e8c-85c8-138613edd7d4\") " pod="opendatahub/maas-controller-8b95c9c66-cv97b" Nov 04 01:14:47.356299 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:47.356161 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/288d4d03-2c1e-4e8c-85c8-138613edd7d4-metrics-certs\") pod \"maas-controller-8b95c9c66-cv97b\" (UID: \"288d4d03-2c1e-4e8c-85c8-138613edd7d4\") " pod="opendatahub/maas-controller-8b95c9c66-cv97b" Nov 04 01:14:47.356299 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:47.356207 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/288d4d03-2c1e-4e8c-85c8-138613edd7d4-webhook-cert\") pod \"maas-controller-8b95c9c66-cv97b\" (UID: \"288d4d03-2c1e-4e8c-85c8-138613edd7d4\") " pod="opendatahub/maas-controller-8b95c9c66-cv97b" Nov 04 01:14:47.359025 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:47.358992 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/288d4d03-2c1e-4e8c-85c8-138613edd7d4-webhook-cert\") pod \"maas-controller-8b95c9c66-cv97b\" (UID: \"288d4d03-2c1e-4e8c-85c8-138613edd7d4\") " pod="opendatahub/maas-controller-8b95c9c66-cv97b" Nov 04 01:14:47.359133 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:47.358992 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/288d4d03-2c1e-4e8c-85c8-138613edd7d4-metrics-certs\") pod \"maas-controller-8b95c9c66-cv97b\" (UID: \"288d4d03-2c1e-4e8c-85c8-138613edd7d4\") " pod="opendatahub/maas-controller-8b95c9c66-cv97b" Nov 04 01:14:47.364195 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:47.364171 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8498f949d-mgqrt"] Nov 04 01:14:47.372569 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:47.372547 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj8v5\" (UniqueName: \"kubernetes.io/projected/288d4d03-2c1e-4e8c-85c8-138613edd7d4-kube-api-access-vj8v5\") pod \"maas-controller-8b95c9c66-cv97b\" (UID: \"288d4d03-2c1e-4e8c-85c8-138613edd7d4\") " pod="opendatahub/maas-controller-8b95c9c66-cv97b" Nov 04 01:14:47.394766 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:47.394735 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8498f949d-mgqrt"] Nov 04 01:14:47.394880 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:47.394855 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8498f949d-mgqrt" Nov 04 01:14:47.456813 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:47.456785 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69707113-e314-4ee5-9e00-0fd663820d62-webhook-cert\") pod \"maas-controller-8498f949d-mgqrt\" (UID: \"69707113-e314-4ee5-9e00-0fd663820d62\") " pod="opendatahub/maas-controller-8498f949d-mgqrt" Nov 04 01:14:47.456974 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:47.456820 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69707113-e314-4ee5-9e00-0fd663820d62-metrics-certs\") pod \"maas-controller-8498f949d-mgqrt\" (UID: \"69707113-e314-4ee5-9e00-0fd663820d62\") " pod="opendatahub/maas-controller-8498f949d-mgqrt" Nov 04 01:14:47.456974 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:47.456848 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2nc9\" (UniqueName: \"kubernetes.io/projected/69707113-e314-4ee5-9e00-0fd663820d62-kube-api-access-s2nc9\") pod \"maas-controller-8498f949d-mgqrt\" (UID: \"69707113-e314-4ee5-9e00-0fd663820d62\") " pod="opendatahub/maas-controller-8498f949d-mgqrt" Nov 04 01:14:47.480720 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:47.480699 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8b95c9c66-cv97b" Nov 04 01:14:47.557733 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:47.557702 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69707113-e314-4ee5-9e00-0fd663820d62-webhook-cert\") pod \"maas-controller-8498f949d-mgqrt\" (UID: \"69707113-e314-4ee5-9e00-0fd663820d62\") " pod="opendatahub/maas-controller-8498f949d-mgqrt" Nov 04 01:14:47.558351 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:47.557951 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69707113-e314-4ee5-9e00-0fd663820d62-metrics-certs\") pod \"maas-controller-8498f949d-mgqrt\" (UID: \"69707113-e314-4ee5-9e00-0fd663820d62\") " pod="opendatahub/maas-controller-8498f949d-mgqrt" Nov 04 01:14:47.558351 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:47.558028 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2nc9\" (UniqueName: \"kubernetes.io/projected/69707113-e314-4ee5-9e00-0fd663820d62-kube-api-access-s2nc9\") pod \"maas-controller-8498f949d-mgqrt\" (UID: \"69707113-e314-4ee5-9e00-0fd663820d62\") " pod="opendatahub/maas-controller-8498f949d-mgqrt" Nov 04 01:14:47.560474 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:47.560430 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69707113-e314-4ee5-9e00-0fd663820d62-metrics-certs\") pod \"maas-controller-8498f949d-mgqrt\" (UID: \"69707113-e314-4ee5-9e00-0fd663820d62\") " pod="opendatahub/maas-controller-8498f949d-mgqrt" Nov 04 01:14:47.560562 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:47.560511 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69707113-e314-4ee5-9e00-0fd663820d62-webhook-cert\") pod \"maas-controller-8498f949d-mgqrt\" (UID: \"69707113-e314-4ee5-9e00-0fd663820d62\") " pod="opendatahub/maas-controller-8498f949d-mgqrt" Nov 04 01:14:47.570409 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:47.570388 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2nc9\" (UniqueName: \"kubernetes.io/projected/69707113-e314-4ee5-9e00-0fd663820d62-kube-api-access-s2nc9\") pod \"maas-controller-8498f949d-mgqrt\" (UID: \"69707113-e314-4ee5-9e00-0fd663820d62\") " pod="opendatahub/maas-controller-8498f949d-mgqrt" Nov 04 01:14:47.712646 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:47.712618 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8498f949d-mgqrt" Nov 04 01:14:47.791733 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:47.791477 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8b95c9c66-cv97b"] Nov 04 01:14:47.796239 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:14:47.796199 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod288d4d03_2c1e_4e8c_85c8_138613edd7d4.slice/crio-cf78dfceffdc45ea9af2deeb68a5d55cff349adda4623c4257ab9f9ec8ba5b6c WatchSource:0}: Error finding container cf78dfceffdc45ea9af2deeb68a5d55cff349adda4623c4257ab9f9ec8ba5b6c: Status 404 returned error can't find the container with id cf78dfceffdc45ea9af2deeb68a5d55cff349adda4623c4257ab9f9ec8ba5b6c Nov 04 01:14:47.934549 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:47.934515 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8b95c9c66-cv97b" event={"ID":"288d4d03-2c1e-4e8c-85c8-138613edd7d4","Type":"ContainerStarted","Data":"cf78dfceffdc45ea9af2deeb68a5d55cff349adda4623c4257ab9f9ec8ba5b6c"} Nov 04 01:14:48.015312 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:48.015285 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8498f949d-mgqrt"] Nov 04 01:14:48.017255 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:14:48.017220 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69707113_e314_4ee5_9e00_0fd663820d62.slice/crio-10842f4b7de158085dd815fac1cc3f15ddf77530777608e12bf13e4a7919ce76 WatchSource:0}: Error finding container 10842f4b7de158085dd815fac1cc3f15ddf77530777608e12bf13e4a7919ce76: Status 404 returned error can't find the container with id 10842f4b7de158085dd815fac1cc3f15ddf77530777608e12bf13e4a7919ce76 Nov 04 01:14:48.946205 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:48.946079 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8498f949d-mgqrt" event={"ID":"69707113-e314-4ee5-9e00-0fd663820d62","Type":"ContainerStarted","Data":"10842f4b7de158085dd815fac1cc3f15ddf77530777608e12bf13e4a7919ce76"} Nov 04 01:14:50.965788 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:50.965748 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8b95c9c66-cv97b" event={"ID":"288d4d03-2c1e-4e8c-85c8-138613edd7d4","Type":"ContainerStarted","Data":"709a5db87b6b9891a5462cb7af2fbbebbd361bde3e40c8c35b8a9df582d60d82"} Nov 04 01:14:50.966296 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:50.966053 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8b95c9c66-cv97b" Nov 04 01:14:50.993736 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:50.993689 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-8b95c9c66-cv97b" podStartSLOduration=0.966561622 podStartE2EDuration="3.993668534s" totalImagesPullingTime="3.027106912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:14:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:14:50.993131443 +0000 UTC m=+975.934980348" watchObservedRunningTime="2026-11-04 01:14:50.993668534 +0000 UTC m=+975.935517437" Nov 04 01:14:51.975907 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:51.975864 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8498f949d-mgqrt" event={"ID":"69707113-e314-4ee5-9e00-0fd663820d62","Type":"ContainerStarted","Data":"a0c4df770db604bb9c4d793e81877c4fa039e35623300fc2d04abb3b8a843267"} Nov 04 01:14:51.976398 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:51.976209 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8498f949d-mgqrt" Nov 04 01:14:52.001408 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:52.001357 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-8498f949d-mgqrt" podStartSLOduration=2.045901597 podStartE2EDuration="5.001344119s" totalImagesPullingTime="2.955442522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:14:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:14:51.999149317 +0000 UTC m=+976.940998221" watchObservedRunningTime="2026-11-04 01:14:52.001344119 +0000 UTC m=+976.943193040" Nov 04 01:14:52.569050 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:52.569014 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-h4j6x"] Nov 04 01:14:52.572685 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:52.572662 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-h4j6x" Nov 04 01:14:52.577203 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:52.577171 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 04 01:14:52.579287 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:52.579180 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 04 01:14:52.579450 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:52.579317 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-swnkf\"" Nov 04 01:14:52.579686 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:52.579669 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 04 01:14:52.579832 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:52.579806 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-h4j6x"] Nov 04 01:14:52.712155 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:52.712127 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c37edcff-4510-449e-a03e-ff4be0250b41-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-h4j6x\" (UID: \"c37edcff-4510-449e-a03e-ff4be0250b41\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-h4j6x" Nov 04 01:14:52.712342 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:52.712191 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c37edcff-4510-449e-a03e-ff4be0250b41-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-h4j6x\" (UID: \"c37edcff-4510-449e-a03e-ff4be0250b41\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-h4j6x" Nov 04 01:14:52.712342 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:52.712287 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c37edcff-4510-449e-a03e-ff4be0250b41-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-h4j6x\" (UID: \"c37edcff-4510-449e-a03e-ff4be0250b41\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-h4j6x" Nov 04 01:14:52.712342 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:52.712308 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx8mw\" (UniqueName: \"kubernetes.io/projected/c37edcff-4510-449e-a03e-ff4be0250b41-kube-api-access-zx8mw\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-h4j6x\" (UID: \"c37edcff-4510-449e-a03e-ff4be0250b41\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-h4j6x" Nov 04 01:14:52.813011 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:52.812974 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c37edcff-4510-449e-a03e-ff4be0250b41-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-h4j6x\" (UID: \"c37edcff-4510-449e-a03e-ff4be0250b41\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-h4j6x" Nov 04 01:14:52.813192 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:52.813054 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c37edcff-4510-449e-a03e-ff4be0250b41-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-h4j6x\" (UID: \"c37edcff-4510-449e-a03e-ff4be0250b41\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-h4j6x" Nov 04 01:14:52.813192 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:52.813079 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx8mw\" (UniqueName: \"kubernetes.io/projected/c37edcff-4510-449e-a03e-ff4be0250b41-kube-api-access-zx8mw\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-h4j6x\" (UID: \"c37edcff-4510-449e-a03e-ff4be0250b41\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-h4j6x" Nov 04 01:14:52.813192 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:52.813126 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c37edcff-4510-449e-a03e-ff4be0250b41-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-h4j6x\" (UID: \"c37edcff-4510-449e-a03e-ff4be0250b41\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-h4j6x" Nov 04 01:14:52.813769 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:52.813742 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c37edcff-4510-449e-a03e-ff4be0250b41-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-h4j6x\" (UID: \"c37edcff-4510-449e-a03e-ff4be0250b41\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-h4j6x" Nov 04 01:14:52.813977 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:52.813958 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c37edcff-4510-449e-a03e-ff4be0250b41-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-h4j6x\" (UID: \"c37edcff-4510-449e-a03e-ff4be0250b41\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-h4j6x" Nov 04 01:14:52.815531 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:52.815515 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c37edcff-4510-449e-a03e-ff4be0250b41-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-h4j6x\" (UID: \"c37edcff-4510-449e-a03e-ff4be0250b41\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-h4j6x" Nov 04 01:14:52.821307 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:52.821258 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx8mw\" (UniqueName: \"kubernetes.io/projected/c37edcff-4510-449e-a03e-ff4be0250b41-kube-api-access-zx8mw\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-h4j6x\" (UID: \"c37edcff-4510-449e-a03e-ff4be0250b41\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-h4j6x" Nov 04 01:14:52.896158 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:52.896127 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-h4j6x" Nov 04 01:14:53.169564 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:53.169540 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-h4j6x"] Nov 04 01:14:53.171944 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:14:53.171901 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc37edcff_4510_449e_a03e_ff4be0250b41.slice/crio-47feb9a6731f0715fd9f5ed653e05e1d0fc2ba896e59047490c8f9812353c92b WatchSource:0}: Error finding container 47feb9a6731f0715fd9f5ed653e05e1d0fc2ba896e59047490c8f9812353c92b: Status 404 returned error can't find the container with id 47feb9a6731f0715fd9f5ed653e05e1d0fc2ba896e59047490c8f9812353c92b Nov 04 01:14:53.987683 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:14:53.987647 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-h4j6x" event={"ID":"c37edcff-4510-449e-a03e-ff4be0250b41","Type":"ContainerStarted","Data":"47feb9a6731f0715fd9f5ed653e05e1d0fc2ba896e59047490c8f9812353c92b"} Nov 04 01:15:01.979274 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:01.979240 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8b95c9c66-cv97b" Nov 04 01:15:02.984585 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:02.984559 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8498f949d-mgqrt" Nov 04 01:15:03.033453 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:03.033410 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8b95c9c66-cv97b"] Nov 04 01:15:03.033701 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:03.033651 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-8b95c9c66-cv97b" podUID="288d4d03-2c1e-4e8c-85c8-138613edd7d4" containerName="manager" containerID="cri-o://709a5db87b6b9891a5462cb7af2fbbebbd361bde3e40c8c35b8a9df582d60d82" gracePeriod=30 Nov 04 01:15:03.304969 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:03.304916 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8b95c9c66-cv97b" Nov 04 01:15:03.416951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:03.416896 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/288d4d03-2c1e-4e8c-85c8-138613edd7d4-kube-api-access-vj8v5\" (UniqueName: \"kubernetes.io/projected/288d4d03-2c1e-4e8c-85c8-138613edd7d4-kube-api-access-vj8v5\") pod \"288d4d03-2c1e-4e8c-85c8-138613edd7d4\" (UID: \"288d4d03-2c1e-4e8c-85c8-138613edd7d4\") " Nov 04 01:15:03.417126 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:03.416956 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/288d4d03-2c1e-4e8c-85c8-138613edd7d4-webhook-cert\" (UniqueName: \"kubernetes.io/secret/288d4d03-2c1e-4e8c-85c8-138613edd7d4-webhook-cert\") pod \"288d4d03-2c1e-4e8c-85c8-138613edd7d4\" (UID: \"288d4d03-2c1e-4e8c-85c8-138613edd7d4\") " Nov 04 01:15:03.417126 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:03.416993 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/288d4d03-2c1e-4e8c-85c8-138613edd7d4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/288d4d03-2c1e-4e8c-85c8-138613edd7d4-metrics-certs\") pod \"288d4d03-2c1e-4e8c-85c8-138613edd7d4\" (UID: \"288d4d03-2c1e-4e8c-85c8-138613edd7d4\") " Nov 04 01:15:03.419690 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:03.419654 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/288d4d03-2c1e-4e8c-85c8-138613edd7d4-metrics-certs" pod "288d4d03-2c1e-4e8c-85c8-138613edd7d4" (UID: "288d4d03-2c1e-4e8c-85c8-138613edd7d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:15:03.419820 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:03.419692 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/288d4d03-2c1e-4e8c-85c8-138613edd7d4-kube-api-access-vj8v5" pod "288d4d03-2c1e-4e8c-85c8-138613edd7d4" (UID: "288d4d03-2c1e-4e8c-85c8-138613edd7d4"). InnerVolumeSpecName "kube-api-access-vj8v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:15:03.419820 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:03.419771 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/288d4d03-2c1e-4e8c-85c8-138613edd7d4-webhook-cert" pod "288d4d03-2c1e-4e8c-85c8-138613edd7d4" (UID: "288d4d03-2c1e-4e8c-85c8-138613edd7d4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:15:03.518200 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:03.518163 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj8v5\" (UniqueName: \"kubernetes.io/projected/288d4d03-2c1e-4e8c-85c8-138613edd7d4-kube-api-access-vj8v5\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:03.518200 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:03.518205 7059 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/288d4d03-2c1e-4e8c-85c8-138613edd7d4-webhook-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:03.518414 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:03.518219 7059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/288d4d03-2c1e-4e8c-85c8-138613edd7d4-metrics-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:04.054294 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:04.054251 7059 generic.go:356] "Generic (PLEG): container finished" podID="288d4d03-2c1e-4e8c-85c8-138613edd7d4" containerID="709a5db87b6b9891a5462cb7af2fbbebbd361bde3e40c8c35b8a9df582d60d82" exitCode=0 Nov 04 01:15:04.054753 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:04.054301 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8b95c9c66-cv97b" Nov 04 01:15:04.054753 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:04.054318 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8b95c9c66-cv97b" event={"ID":"288d4d03-2c1e-4e8c-85c8-138613edd7d4","Type":"ContainerDied","Data":"709a5db87b6b9891a5462cb7af2fbbebbd361bde3e40c8c35b8a9df582d60d82"} Nov 04 01:15:04.054753 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:04.054371 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8b95c9c66-cv97b" event={"ID":"288d4d03-2c1e-4e8c-85c8-138613edd7d4","Type":"ContainerDied","Data":"cf78dfceffdc45ea9af2deeb68a5d55cff349adda4623c4257ab9f9ec8ba5b6c"} Nov 04 01:15:04.054753 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:04.054400 7059 scope.go:122] "RemoveContainer" containerID="709a5db87b6b9891a5462cb7af2fbbebbd361bde3e40c8c35b8a9df582d60d82" Nov 04 01:15:04.064785 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:04.064756 7059 scope.go:122] "RemoveContainer" containerID="709a5db87b6b9891a5462cb7af2fbbebbd361bde3e40c8c35b8a9df582d60d82" Nov 04 01:15:04.065162 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:15:04.065126 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"709a5db87b6b9891a5462cb7af2fbbebbd361bde3e40c8c35b8a9df582d60d82\": container with ID starting with 709a5db87b6b9891a5462cb7af2fbbebbd361bde3e40c8c35b8a9df582d60d82 not found: ID does not exist" containerID="709a5db87b6b9891a5462cb7af2fbbebbd361bde3e40c8c35b8a9df582d60d82" Nov 04 01:15:04.065248 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:04.065176 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"709a5db87b6b9891a5462cb7af2fbbebbd361bde3e40c8c35b8a9df582d60d82"} err="failed to get container status \"709a5db87b6b9891a5462cb7af2fbbebbd361bde3e40c8c35b8a9df582d60d82\": rpc error: code = NotFound desc = could not find container \"709a5db87b6b9891a5462cb7af2fbbebbd361bde3e40c8c35b8a9df582d60d82\": container with ID starting with 709a5db87b6b9891a5462cb7af2fbbebbd361bde3e40c8c35b8a9df582d60d82 not found: ID does not exist" Nov 04 01:15:04.077262 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:04.077236 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8b95c9c66-cv97b"] Nov 04 01:15:04.081106 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:04.081068 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8b95c9c66-cv97b"] Nov 04 01:15:05.174902 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:05.174874 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="288d4d03-2c1e-4e8c-85c8-138613edd7d4" path="/var/lib/kubelet/pods/288d4d03-2c1e-4e8c-85c8-138613edd7d4/volumes" Nov 04 01:15:06.066966 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:06.066912 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-h4j6x" event={"ID":"c37edcff-4510-449e-a03e-ff4be0250b41","Type":"ContainerStarted","Data":"f267488eb145ac304bc92cd3d8784b9985125c9c9de2b4e0b03cd1b8a041e452"} Nov 04 01:15:06.067142 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:06.067067 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-h4j6x" Nov 04 01:15:06.084249 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:06.084182 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-h4j6x" podStartSLOduration=2.245200719 podStartE2EDuration="14.084170136s" totalImagesPullingTime="11.838969417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:06.082875725 +0000 UTC m=+991.024724629" watchObservedRunningTime="2026-11-04 01:15:06.084170136 +0000 UTC m=+991.026019038" Nov 04 01:15:06.478986 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:15:06.478954 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:15:17.077575 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:17.077494 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-h4j6x" Nov 04 01:15:22.771417 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:22.771383 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-674d648f7b-pzjgb"] Nov 04 01:15:22.771869 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:22.771762 7059 cpu_manager.go:414] "containerMap: removing container" podUID="288d4d03-2c1e-4e8c-85c8-138613edd7d4" containerName="manager" Nov 04 01:15:22.771869 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:22.771773 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="288d4d03-2c1e-4e8c-85c8-138613edd7d4" containerName="manager" Nov 04 01:15:22.771869 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:22.771841 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="288d4d03-2c1e-4e8c-85c8-138613edd7d4" containerName="manager" Nov 04 01:15:22.775049 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:22.775023 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-674d648f7b-pzjgb" Nov 04 01:15:22.782314 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:22.781971 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db5bb392-a03b-4ea9-93bf-4bf25fb3fddc-maas-api-tls\") pod \"maas-api-674d648f7b-pzjgb\" (UID: \"db5bb392-a03b-4ea9-93bf-4bf25fb3fddc\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-pzjgb" Nov 04 01:15:22.782314 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:22.782016 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r456g\" (UniqueName: \"kubernetes.io/projected/db5bb392-a03b-4ea9-93bf-4bf25fb3fddc-kube-api-access-r456g\") pod \"maas-api-674d648f7b-pzjgb\" (UID: \"db5bb392-a03b-4ea9-93bf-4bf25fb3fddc\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-pzjgb" Nov 04 01:15:22.782314 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:22.782045 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db5bb392-a03b-4ea9-93bf-4bf25fb3fddc-metrics-certs\") pod \"maas-api-674d648f7b-pzjgb\" (UID: \"db5bb392-a03b-4ea9-93bf-4bf25fb3fddc\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-pzjgb" Nov 04 01:15:22.782314 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:22.782089 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5bb392-a03b-4ea9-93bf-4bf25fb3fddc-openshift-service-ca\") pod \"maas-api-674d648f7b-pzjgb\" (UID: \"db5bb392-a03b-4ea9-93bf-4bf25fb3fddc\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-pzjgb" Nov 04 01:15:22.782972 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:22.782939 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 04 01:15:22.783106 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:22.783038 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 04 01:15:22.784358 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:22.784337 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nb79d\"" Nov 04 01:15:22.786340 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:22.786320 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-674d648f7b-pzjgb"] Nov 04 01:15:22.883231 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:22.883199 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5bb392-a03b-4ea9-93bf-4bf25fb3fddc-openshift-service-ca\") pod \"maas-api-674d648f7b-pzjgb\" (UID: \"db5bb392-a03b-4ea9-93bf-4bf25fb3fddc\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-pzjgb" Nov 04 01:15:22.883426 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:22.883294 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db5bb392-a03b-4ea9-93bf-4bf25fb3fddc-maas-api-tls\") pod \"maas-api-674d648f7b-pzjgb\" (UID: \"db5bb392-a03b-4ea9-93bf-4bf25fb3fddc\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-pzjgb" Nov 04 01:15:22.883426 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:22.883316 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r456g\" (UniqueName: \"kubernetes.io/projected/db5bb392-a03b-4ea9-93bf-4bf25fb3fddc-kube-api-access-r456g\") pod \"maas-api-674d648f7b-pzjgb\" (UID: \"db5bb392-a03b-4ea9-93bf-4bf25fb3fddc\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-pzjgb" Nov 04 01:15:22.883426 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:22.883330 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db5bb392-a03b-4ea9-93bf-4bf25fb3fddc-metrics-certs\") pod \"maas-api-674d648f7b-pzjgb\" (UID: \"db5bb392-a03b-4ea9-93bf-4bf25fb3fddc\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-pzjgb" Nov 04 01:15:22.883914 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:22.883891 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5bb392-a03b-4ea9-93bf-4bf25fb3fddc-openshift-service-ca\") pod \"maas-api-674d648f7b-pzjgb\" (UID: \"db5bb392-a03b-4ea9-93bf-4bf25fb3fddc\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-pzjgb" Nov 04 01:15:22.885950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:22.885912 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db5bb392-a03b-4ea9-93bf-4bf25fb3fddc-metrics-certs\") pod \"maas-api-674d648f7b-pzjgb\" (UID: \"db5bb392-a03b-4ea9-93bf-4bf25fb3fddc\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-pzjgb" Nov 04 01:15:22.886028 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:22.886009 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db5bb392-a03b-4ea9-93bf-4bf25fb3fddc-maas-api-tls\") pod \"maas-api-674d648f7b-pzjgb\" (UID: \"db5bb392-a03b-4ea9-93bf-4bf25fb3fddc\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-pzjgb" Nov 04 01:15:22.891589 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:22.891564 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r456g\" (UniqueName: \"kubernetes.io/projected/db5bb392-a03b-4ea9-93bf-4bf25fb3fddc-kube-api-access-r456g\") pod \"maas-api-674d648f7b-pzjgb\" (UID: \"db5bb392-a03b-4ea9-93bf-4bf25fb3fddc\") " pod="odh-ai-gateway-infra/maas-api-674d648f7b-pzjgb" Nov 04 01:15:23.090160 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:23.090081 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-674d648f7b-pzjgb" Nov 04 01:15:23.371466 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:23.371440 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-674d648f7b-pzjgb"] Nov 04 01:15:23.373845 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:15:23.373809 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb5bb392_a03b_4ea9_93bf_4bf25fb3fddc.slice/crio-e198285264f7d28bb367994d05426cff7bf4fe18c3da6af53c02dff0a5d76d5f WatchSource:0}: Error finding container e198285264f7d28bb367994d05426cff7bf4fe18c3da6af53c02dff0a5d76d5f: Status 404 returned error can't find the container with id e198285264f7d28bb367994d05426cff7bf4fe18c3da6af53c02dff0a5d76d5f Nov 04 01:15:24.158448 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:24.158414 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-674d648f7b-pzjgb" event={"ID":"db5bb392-a03b-4ea9-93bf-4bf25fb3fddc","Type":"ContainerStarted","Data":"e198285264f7d28bb367994d05426cff7bf4fe18c3da6af53c02dff0a5d76d5f"} Nov 04 01:15:26.169003 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:26.168961 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-674d648f7b-pzjgb" event={"ID":"db5bb392-a03b-4ea9-93bf-4bf25fb3fddc","Type":"ContainerStarted","Data":"3adfeaa78510fdd7f0e2b91769ab10892ac61ca9af13f99bda50598f0e7f86c5"} Nov 04 01:15:26.169425 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:26.169099 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-674d648f7b-pzjgb" Nov 04 01:15:26.187099 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:26.187049 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-674d648f7b-pzjgb" podStartSLOduration=2.202518112 podStartE2EDuration="4.187034961s" totalImagesPullingTime="1.984516849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:26.185730528 +0000 UTC m=+1011.127579457" watchObservedRunningTime="2026-11-04 01:15:26.187034961 +0000 UTC m=+1011.128883865" Nov 04 01:15:32.180218 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:32.180191 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-674d648f7b-pzjgb" Nov 04 01:15:32.864070 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:32.864036 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6dd76875b5-nrnkh"] Nov 04 01:15:33.144309 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:33.144229 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-nrnkh"] Nov 04 01:15:33.144309 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:33.144265 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-nrnkh"] Nov 04 01:15:33.144481 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:33.144330 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-nrnkh" Nov 04 01:15:33.164917 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:33.164888 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-nrnkh" Nov 04 01:15:33.203486 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:33.203458 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-nrnkh" Nov 04 01:15:33.233095 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:33.233062 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-nrnkh"] Nov 04 01:15:33.241123 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:33.241097 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-nrnkh"] Nov 04 01:15:33.297596 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:33.297569 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-648b7c5784-dn7sb"] Nov 04 01:15:33.322615 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:33.322589 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-648b7c5784-dn7sb"] Nov 04 01:15:33.322747 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:33.322594 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-648b7c5784-dn7sb" Nov 04 01:15:33.326108 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:33.326090 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 04 01:15:33.326236 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:33.326126 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2xvwz\"" Nov 04 01:15:33.372796 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:33.372767 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l24xd\" (UniqueName: \"kubernetes.io/projected/bd2407fe-d5a7-4207-8e0a-1bc1980abeb0-kube-api-access-l24xd\") pod \"authorino-648b7c5784-dn7sb\" (UID: \"bd2407fe-d5a7-4207-8e0a-1bc1980abeb0\") " pod="kuadrant-system/authorino-648b7c5784-dn7sb" Nov 04 01:15:33.372904 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:33.372813 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bd2407fe-d5a7-4207-8e0a-1bc1980abeb0-tls-cert\") pod \"authorino-648b7c5784-dn7sb\" (UID: \"bd2407fe-d5a7-4207-8e0a-1bc1980abeb0\") " pod="kuadrant-system/authorino-648b7c5784-dn7sb" Nov 04 01:15:33.473859 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:33.473836 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l24xd\" (UniqueName: \"kubernetes.io/projected/bd2407fe-d5a7-4207-8e0a-1bc1980abeb0-kube-api-access-l24xd\") pod \"authorino-648b7c5784-dn7sb\" (UID: \"bd2407fe-d5a7-4207-8e0a-1bc1980abeb0\") " pod="kuadrant-system/authorino-648b7c5784-dn7sb" Nov 04 01:15:33.474014 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:33.473881 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bd2407fe-d5a7-4207-8e0a-1bc1980abeb0-tls-cert\") pod \"authorino-648b7c5784-dn7sb\" (UID: \"bd2407fe-d5a7-4207-8e0a-1bc1980abeb0\") " pod="kuadrant-system/authorino-648b7c5784-dn7sb" Nov 04 01:15:33.476210 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:33.476191 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bd2407fe-d5a7-4207-8e0a-1bc1980abeb0-tls-cert\") pod \"authorino-648b7c5784-dn7sb\" (UID: \"bd2407fe-d5a7-4207-8e0a-1bc1980abeb0\") " pod="kuadrant-system/authorino-648b7c5784-dn7sb" Nov 04 01:15:33.482099 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:33.482079 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l24xd\" (UniqueName: \"kubernetes.io/projected/bd2407fe-d5a7-4207-8e0a-1bc1980abeb0-kube-api-access-l24xd\") pod \"authorino-648b7c5784-dn7sb\" (UID: \"bd2407fe-d5a7-4207-8e0a-1bc1980abeb0\") " pod="kuadrant-system/authorino-648b7c5784-dn7sb" Nov 04 01:15:33.645352 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:33.645323 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-648b7c5784-dn7sb" Nov 04 01:15:33.915117 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:33.915097 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-648b7c5784-dn7sb"] Nov 04 01:15:33.917530 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:15:33.917503 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd2407fe_d5a7_4207_8e0a_1bc1980abeb0.slice/crio-e048003e48a6fe2a4a4b9973ca7414949909442a956a5b087072f71cdaec5f6d WatchSource:0}: Error finding container e048003e48a6fe2a4a4b9973ca7414949909442a956a5b087072f71cdaec5f6d: Status 404 returned error can't find the container with id e048003e48a6fe2a4a4b9973ca7414949909442a956a5b087072f71cdaec5f6d Nov 04 01:15:34.209140 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:34.209108 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-648b7c5784-dn7sb" event={"ID":"bd2407fe-d5a7-4207-8e0a-1bc1980abeb0","Type":"ContainerStarted","Data":"e048003e48a6fe2a4a4b9973ca7414949909442a956a5b087072f71cdaec5f6d"} Nov 04 01:15:35.216976 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:35.216939 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-648b7c5784-dn7sb" event={"ID":"bd2407fe-d5a7-4207-8e0a-1bc1980abeb0","Type":"ContainerStarted","Data":"eb4939e7b4f6b726ebc48a0c3fd06c2b38fb61f2ed5d4db9a2a2979d6a9faa64"} Nov 04 01:15:35.232514 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:35.232472 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-648b7c5784-dn7sb" podStartSLOduration=1.493465891 podStartE2EDuration="2.232460034s" totalImagesPullingTime="738.994143ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:35.230908054 +0000 UTC m=+1020.172756957" watchObservedRunningTime="2026-11-04 01:15:35.232460034 +0000 UTC m=+1020.174308916" Nov 04 01:15:36.478649 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:15:36.478614 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:15:42.887055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:42.887021 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-674d648f7b-pzjgb"] Nov 04 01:15:42.887438 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:42.887231 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-674d648f7b-pzjgb" podUID="db5bb392-a03b-4ea9-93bf-4bf25fb3fddc" containerName="maas-api" containerID="cri-o://3adfeaa78510fdd7f0e2b91769ab10892ac61ca9af13f99bda50598f0e7f86c5" gracePeriod=30 Nov 04 01:15:43.122788 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:43.122765 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-674d648f7b-pzjgb" Nov 04 01:15:43.254166 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:43.254140 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db5bb392-a03b-4ea9-93bf-4bf25fb3fddc-kube-api-access-r456g\" (UniqueName: \"kubernetes.io/projected/db5bb392-a03b-4ea9-93bf-4bf25fb3fddc-kube-api-access-r456g\") pod \"db5bb392-a03b-4ea9-93bf-4bf25fb3fddc\" (UID: \"db5bb392-a03b-4ea9-93bf-4bf25fb3fddc\") " Nov 04 01:15:43.254329 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:43.254234 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db5bb392-a03b-4ea9-93bf-4bf25fb3fddc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5bb392-a03b-4ea9-93bf-4bf25fb3fddc-openshift-service-ca\") pod \"db5bb392-a03b-4ea9-93bf-4bf25fb3fddc\" (UID: \"db5bb392-a03b-4ea9-93bf-4bf25fb3fddc\") " Nov 04 01:15:43.254329 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:43.254266 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db5bb392-a03b-4ea9-93bf-4bf25fb3fddc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db5bb392-a03b-4ea9-93bf-4bf25fb3fddc-maas-api-tls\") pod \"db5bb392-a03b-4ea9-93bf-4bf25fb3fddc\" (UID: \"db5bb392-a03b-4ea9-93bf-4bf25fb3fddc\") " Nov 04 01:15:43.254329 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:43.254294 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db5bb392-a03b-4ea9-93bf-4bf25fb3fddc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/db5bb392-a03b-4ea9-93bf-4bf25fb3fddc-metrics-certs\") pod \"db5bb392-a03b-4ea9-93bf-4bf25fb3fddc\" (UID: \"db5bb392-a03b-4ea9-93bf-4bf25fb3fddc\") " Nov 04 01:15:43.254590 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:43.254565 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db5bb392-a03b-4ea9-93bf-4bf25fb3fddc-openshift-service-ca" pod "db5bb392-a03b-4ea9-93bf-4bf25fb3fddc" (UID: "db5bb392-a03b-4ea9-93bf-4bf25fb3fddc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:15:43.256596 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:43.256567 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db5bb392-a03b-4ea9-93bf-4bf25fb3fddc-metrics-certs" pod "db5bb392-a03b-4ea9-93bf-4bf25fb3fddc" (UID: "db5bb392-a03b-4ea9-93bf-4bf25fb3fddc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:15:43.256703 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:43.256641 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db5bb392-a03b-4ea9-93bf-4bf25fb3fddc-maas-api-tls" pod "db5bb392-a03b-4ea9-93bf-4bf25fb3fddc" (UID: "db5bb392-a03b-4ea9-93bf-4bf25fb3fddc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:15:43.256769 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:43.256713 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db5bb392-a03b-4ea9-93bf-4bf25fb3fddc-kube-api-access-r456g" pod "db5bb392-a03b-4ea9-93bf-4bf25fb3fddc" (UID: "db5bb392-a03b-4ea9-93bf-4bf25fb3fddc"). InnerVolumeSpecName "kube-api-access-r456g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:15:43.257627 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:43.257607 7059 generic.go:356] "Generic (PLEG): container finished" podID="db5bb392-a03b-4ea9-93bf-4bf25fb3fddc" containerID="3adfeaa78510fdd7f0e2b91769ab10892ac61ca9af13f99bda50598f0e7f86c5" exitCode=0 Nov 04 01:15:43.257710 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:43.257653 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-674d648f7b-pzjgb" Nov 04 01:15:43.257710 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:43.257664 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-674d648f7b-pzjgb" event={"ID":"db5bb392-a03b-4ea9-93bf-4bf25fb3fddc","Type":"ContainerDied","Data":"3adfeaa78510fdd7f0e2b91769ab10892ac61ca9af13f99bda50598f0e7f86c5"} Nov 04 01:15:43.257710 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:43.257706 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-674d648f7b-pzjgb" event={"ID":"db5bb392-a03b-4ea9-93bf-4bf25fb3fddc","Type":"ContainerDied","Data":"e198285264f7d28bb367994d05426cff7bf4fe18c3da6af53c02dff0a5d76d5f"} Nov 04 01:15:43.257806 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:43.257728 7059 scope.go:122] "RemoveContainer" containerID="3adfeaa78510fdd7f0e2b91769ab10892ac61ca9af13f99bda50598f0e7f86c5" Nov 04 01:15:43.279105 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:43.279091 7059 scope.go:122] "RemoveContainer" containerID="3adfeaa78510fdd7f0e2b91769ab10892ac61ca9af13f99bda50598f0e7f86c5" Nov 04 01:15:43.279384 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:15:43.279367 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3adfeaa78510fdd7f0e2b91769ab10892ac61ca9af13f99bda50598f0e7f86c5\": container with ID starting with 3adfeaa78510fdd7f0e2b91769ab10892ac61ca9af13f99bda50598f0e7f86c5 not found: ID does not exist" containerID="3adfeaa78510fdd7f0e2b91769ab10892ac61ca9af13f99bda50598f0e7f86c5" Nov 04 01:15:43.279443 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:43.279396 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3adfeaa78510fdd7f0e2b91769ab10892ac61ca9af13f99bda50598f0e7f86c5"} err="failed to get container status \"3adfeaa78510fdd7f0e2b91769ab10892ac61ca9af13f99bda50598f0e7f86c5\": rpc error: code = NotFound desc = could not find container \"3adfeaa78510fdd7f0e2b91769ab10892ac61ca9af13f99bda50598f0e7f86c5\": container with ID starting with 3adfeaa78510fdd7f0e2b91769ab10892ac61ca9af13f99bda50598f0e7f86c5 not found: ID does not exist" Nov 04 01:15:43.289973 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:43.289903 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-674d648f7b-pzjgb"] Nov 04 01:15:43.291872 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:43.291842 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-674d648f7b-pzjgb"] Nov 04 01:15:43.355321 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:43.355295 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r456g\" (UniqueName: \"kubernetes.io/projected/db5bb392-a03b-4ea9-93bf-4bf25fb3fddc-kube-api-access-r456g\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:43.355410 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:43.355320 7059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5bb392-a03b-4ea9-93bf-4bf25fb3fddc-openshift-service-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:43.355410 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:43.355338 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db5bb392-a03b-4ea9-93bf-4bf25fb3fddc-maas-api-tls\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:43.355410 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:43.355351 7059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db5bb392-a03b-4ea9-93bf-4bf25fb3fddc-metrics-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:45.177178 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:15:45.177137 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db5bb392-a03b-4ea9-93bf-4bf25fb3fddc" path="/var/lib/kubelet/pods/db5bb392-a03b-4ea9-93bf-4bf25fb3fddc/volumes" Nov 04 01:16:06.479111 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:16:06.479074 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:16:10.576410 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.576381 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b"] Nov 04 01:16:10.576790 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.576764 7059 cpu_manager.go:414] "containerMap: removing container" podUID="db5bb392-a03b-4ea9-93bf-4bf25fb3fddc" containerName="maas-api" Nov 04 01:16:10.576790 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.576775 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db5bb392-a03b-4ea9-93bf-4bf25fb3fddc" containerName="maas-api" Nov 04 01:16:10.576865 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.576831 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="db5bb392-a03b-4ea9-93bf-4bf25fb3fddc" containerName="maas-api" Nov 04 01:16:10.580446 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.580427 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" Nov 04 01:16:10.594366 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.594347 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-f28wr\"" Nov 04 01:16:10.600390 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.600365 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b"] Nov 04 01:16:10.676852 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.676825 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/994ce0d7-9fe2-494e-a915-94fe06da822e-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-7rw6b\" (UID: \"994ce0d7-9fe2-494e-a915-94fe06da822e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" Nov 04 01:16:10.677017 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.676858 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/994ce0d7-9fe2-494e-a915-94fe06da822e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-7rw6b\" (UID: \"994ce0d7-9fe2-494e-a915-94fe06da822e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" Nov 04 01:16:10.677017 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.676884 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/994ce0d7-9fe2-494e-a915-94fe06da822e-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-7rw6b\" (UID: \"994ce0d7-9fe2-494e-a915-94fe06da822e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" Nov 04 01:16:10.677017 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.676983 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/994ce0d7-9fe2-494e-a915-94fe06da822e-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-7rw6b\" (UID: \"994ce0d7-9fe2-494e-a915-94fe06da822e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" Nov 04 01:16:10.677163 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.677108 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/994ce0d7-9fe2-494e-a915-94fe06da822e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-7rw6b\" (UID: \"994ce0d7-9fe2-494e-a915-94fe06da822e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" Nov 04 01:16:10.677163 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.677134 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/994ce0d7-9fe2-494e-a915-94fe06da822e-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-7rw6b\" (UID: \"994ce0d7-9fe2-494e-a915-94fe06da822e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" Nov 04 01:16:10.677163 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.677155 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/994ce0d7-9fe2-494e-a915-94fe06da822e-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-7rw6b\" (UID: \"994ce0d7-9fe2-494e-a915-94fe06da822e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" Nov 04 01:16:10.677279 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.677204 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/994ce0d7-9fe2-494e-a915-94fe06da822e-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-7rw6b\" (UID: \"994ce0d7-9fe2-494e-a915-94fe06da822e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" Nov 04 01:16:10.677316 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.677276 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ghh8\" (UniqueName: \"kubernetes.io/projected/994ce0d7-9fe2-494e-a915-94fe06da822e-kube-api-access-9ghh8\") pod \"maas-default-gateway-openshift-default-75d7c785f-7rw6b\" (UID: \"994ce0d7-9fe2-494e-a915-94fe06da822e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" Nov 04 01:16:10.777774 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.777738 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/994ce0d7-9fe2-494e-a915-94fe06da822e-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-7rw6b\" (UID: \"994ce0d7-9fe2-494e-a915-94fe06da822e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" Nov 04 01:16:10.777774 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.777775 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/994ce0d7-9fe2-494e-a915-94fe06da822e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-7rw6b\" (UID: \"994ce0d7-9fe2-494e-a915-94fe06da822e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" Nov 04 01:16:10.778020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.777804 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/994ce0d7-9fe2-494e-a915-94fe06da822e-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-7rw6b\" (UID: \"994ce0d7-9fe2-494e-a915-94fe06da822e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" Nov 04 01:16:10.778020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.777837 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/994ce0d7-9fe2-494e-a915-94fe06da822e-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-7rw6b\" (UID: \"994ce0d7-9fe2-494e-a915-94fe06da822e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" Nov 04 01:16:10.778020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.777900 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/994ce0d7-9fe2-494e-a915-94fe06da822e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-7rw6b\" (UID: \"994ce0d7-9fe2-494e-a915-94fe06da822e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" Nov 04 01:16:10.778020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.777956 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/994ce0d7-9fe2-494e-a915-94fe06da822e-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-7rw6b\" (UID: \"994ce0d7-9fe2-494e-a915-94fe06da822e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" Nov 04 01:16:10.778020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.777988 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/994ce0d7-9fe2-494e-a915-94fe06da822e-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-7rw6b\" (UID: \"994ce0d7-9fe2-494e-a915-94fe06da822e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" Nov 04 01:16:10.778242 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.778035 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/994ce0d7-9fe2-494e-a915-94fe06da822e-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-7rw6b\" (UID: \"994ce0d7-9fe2-494e-a915-94fe06da822e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" Nov 04 01:16:10.778242 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.778061 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ghh8\" (UniqueName: \"kubernetes.io/projected/994ce0d7-9fe2-494e-a915-94fe06da822e-kube-api-access-9ghh8\") pod \"maas-default-gateway-openshift-default-75d7c785f-7rw6b\" (UID: \"994ce0d7-9fe2-494e-a915-94fe06da822e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" Nov 04 01:16:10.778242 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.778213 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/994ce0d7-9fe2-494e-a915-94fe06da822e-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-7rw6b\" (UID: \"994ce0d7-9fe2-494e-a915-94fe06da822e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" Nov 04 01:16:10.778242 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.778227 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/994ce0d7-9fe2-494e-a915-94fe06da822e-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-7rw6b\" (UID: \"994ce0d7-9fe2-494e-a915-94fe06da822e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" Nov 04 01:16:10.778463 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.778398 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/994ce0d7-9fe2-494e-a915-94fe06da822e-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-7rw6b\" (UID: \"994ce0d7-9fe2-494e-a915-94fe06da822e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" Nov 04 01:16:10.778530 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.778473 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/994ce0d7-9fe2-494e-a915-94fe06da822e-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-7rw6b\" (UID: \"994ce0d7-9fe2-494e-a915-94fe06da822e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" Nov 04 01:16:10.778720 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.778701 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/994ce0d7-9fe2-494e-a915-94fe06da822e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-7rw6b\" (UID: \"994ce0d7-9fe2-494e-a915-94fe06da822e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" Nov 04 01:16:10.780495 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.780474 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/994ce0d7-9fe2-494e-a915-94fe06da822e-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-7rw6b\" (UID: \"994ce0d7-9fe2-494e-a915-94fe06da822e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" Nov 04 01:16:10.781237 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.781214 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/994ce0d7-9fe2-494e-a915-94fe06da822e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-7rw6b\" (UID: \"994ce0d7-9fe2-494e-a915-94fe06da822e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" Nov 04 01:16:10.787370 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.787351 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/994ce0d7-9fe2-494e-a915-94fe06da822e-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-7rw6b\" (UID: \"994ce0d7-9fe2-494e-a915-94fe06da822e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" Nov 04 01:16:10.787643 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.787625 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ghh8\" (UniqueName: \"kubernetes.io/projected/994ce0d7-9fe2-494e-a915-94fe06da822e-kube-api-access-9ghh8\") pod \"maas-default-gateway-openshift-default-75d7c785f-7rw6b\" (UID: \"994ce0d7-9fe2-494e-a915-94fe06da822e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" Nov 04 01:16:10.893974 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:10.893864 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" Nov 04 01:16:11.177295 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:11.177268 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b"] Nov 04 01:16:11.181214 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:16:11.181187 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod994ce0d7_9fe2_494e_a915_94fe06da822e.slice/crio-9db656d5821f1445ee88dd8ae52e93888977a28a12c3808e54afbb592b1c1479 WatchSource:0}: Error finding container 9db656d5821f1445ee88dd8ae52e93888977a28a12c3808e54afbb592b1c1479: Status 404 returned error can't find the container with id 9db656d5821f1445ee88dd8ae52e93888977a28a12c3808e54afbb592b1c1479 Nov 04 01:16:11.183242 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:11.183200 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:16:11.183338 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:11.183287 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:16:11.183338 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:11.183330 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:16:11.395588 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:11.395542 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" event={"ID":"994ce0d7-9fe2-494e-a915-94fe06da822e","Type":"ContainerStarted","Data":"36ed63f8e3009912d27fe7d89a7d9010701c8317b5cba61fcc281355acce6974"} Nov 04 01:16:11.395742 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:11.395608 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" event={"ID":"994ce0d7-9fe2-494e-a915-94fe06da822e","Type":"ContainerStarted","Data":"9db656d5821f1445ee88dd8ae52e93888977a28a12c3808e54afbb592b1c1479"} Nov 04 01:16:11.417414 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:11.417364 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" podStartSLOduration=1.417346237 podStartE2EDuration="1.417346237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:11.416461739 +0000 UTC m=+1056.358310645" watchObservedRunningTime="2026-11-04 01:16:11.417346237 +0000 UTC m=+1056.359195139" Nov 04 01:16:11.894840 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:11.894806 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" Nov 04 01:16:12.896199 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:12.896144 7059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-7rw6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 04 01:16:12.896621 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:12.896227 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" podUID="994ce0d7-9fe2-494e-a915-94fe06da822e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 04 01:16:13.895124 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:13.895085 7059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-7rw6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 04 01:16:13.895287 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:13.895164 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" podUID="994ce0d7-9fe2-494e-a915-94fe06da822e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 04 01:16:14.894393 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:14.894350 7059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-7rw6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 04 01:16:14.894772 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:14.894447 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" podUID="994ce0d7-9fe2-494e-a915-94fe06da822e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 04 01:16:15.894695 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:15.894661 7059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-7rw6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 04 01:16:15.895080 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:15.894721 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" podUID="994ce0d7-9fe2-494e-a915-94fe06da822e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 04 01:16:16.895142 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:16.895107 7059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-7rw6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 04 01:16:16.895659 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:16.895171 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" podUID="994ce0d7-9fe2-494e-a915-94fe06da822e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 04 01:16:17.899012 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:17.898982 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" Nov 04 01:16:17.899411 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:17.899265 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" Nov 04 01:16:17.899969 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:17.899947 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" Nov 04 01:16:25.596696 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.596659 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn"] Nov 04 01:16:25.601153 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.601128 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" Nov 04 01:16:25.618630 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.618603 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn"] Nov 04 01:16:25.625290 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.625268 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x"] Nov 04 01:16:25.628806 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.628788 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" Nov 04 01:16:25.649500 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.646764 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x"] Nov 04 01:16:25.649500 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.647649 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26881d61-360e-43bb-83f8-3122e0026e41-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-ddd7x\" (UID: \"26881d61-360e-43bb-83f8-3122e0026e41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" Nov 04 01:16:25.652262 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.652236 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26881d61-360e-43bb-83f8-3122e0026e41-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-ddd7x\" (UID: \"26881d61-360e-43bb-83f8-3122e0026e41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" Nov 04 01:16:25.652465 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.652449 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e65c0d9-95d9-43a6-b77f-bd41b3fd6383-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-bprgn\" (UID: \"5e65c0d9-95d9-43a6-b77f-bd41b3fd6383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" Nov 04 01:16:25.652580 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.652566 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26881d61-360e-43bb-83f8-3122e0026e41-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-ddd7x\" (UID: \"26881d61-360e-43bb-83f8-3122e0026e41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" Nov 04 01:16:25.652844 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.652819 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e65c0d9-95d9-43a6-b77f-bd41b3fd6383-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-bprgn\" (UID: \"5e65c0d9-95d9-43a6-b77f-bd41b3fd6383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" Nov 04 01:16:25.652945 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.652863 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e65c0d9-95d9-43a6-b77f-bd41b3fd6383-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-bprgn\" (UID: \"5e65c0d9-95d9-43a6-b77f-bd41b3fd6383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" Nov 04 01:16:25.652945 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.652906 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e65c0d9-95d9-43a6-b77f-bd41b3fd6383-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-bprgn\" (UID: \"5e65c0d9-95d9-43a6-b77f-bd41b3fd6383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" Nov 04 01:16:25.653110 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.652972 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmwth\" (UniqueName: \"kubernetes.io/projected/26881d61-360e-43bb-83f8-3122e0026e41-kube-api-access-qmwth\") pod \"maas-default-gateway-openshift-default-75d7c785f-ddd7x\" (UID: \"26881d61-360e-43bb-83f8-3122e0026e41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" Nov 04 01:16:25.653110 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.653010 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e65c0d9-95d9-43a6-b77f-bd41b3fd6383-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-bprgn\" (UID: \"5e65c0d9-95d9-43a6-b77f-bd41b3fd6383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" Nov 04 01:16:25.653110 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.653045 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26881d61-360e-43bb-83f8-3122e0026e41-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-ddd7x\" (UID: \"26881d61-360e-43bb-83f8-3122e0026e41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" Nov 04 01:16:25.653110 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.653072 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26881d61-360e-43bb-83f8-3122e0026e41-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-ddd7x\" (UID: \"26881d61-360e-43bb-83f8-3122e0026e41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" Nov 04 01:16:25.653312 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.653109 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26881d61-360e-43bb-83f8-3122e0026e41-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-ddd7x\" (UID: \"26881d61-360e-43bb-83f8-3122e0026e41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" Nov 04 01:16:25.653312 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.653141 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26881d61-360e-43bb-83f8-3122e0026e41-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-ddd7x\" (UID: \"26881d61-360e-43bb-83f8-3122e0026e41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" Nov 04 01:16:25.653312 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.653193 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26881d61-360e-43bb-83f8-3122e0026e41-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-ddd7x\" (UID: \"26881d61-360e-43bb-83f8-3122e0026e41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" Nov 04 01:16:25.754286 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.754259 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e65c0d9-95d9-43a6-b77f-bd41b3fd6383-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-bprgn\" (UID: \"5e65c0d9-95d9-43a6-b77f-bd41b3fd6383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" Nov 04 01:16:25.754415 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.754294 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e65c0d9-95d9-43a6-b77f-bd41b3fd6383-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-bprgn\" (UID: \"5e65c0d9-95d9-43a6-b77f-bd41b3fd6383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" Nov 04 01:16:25.754415 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.754314 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmwth\" (UniqueName: \"kubernetes.io/projected/26881d61-360e-43bb-83f8-3122e0026e41-kube-api-access-qmwth\") pod \"maas-default-gateway-openshift-default-75d7c785f-ddd7x\" (UID: \"26881d61-360e-43bb-83f8-3122e0026e41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" Nov 04 01:16:25.754415 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.754332 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e65c0d9-95d9-43a6-b77f-bd41b3fd6383-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-bprgn\" (UID: \"5e65c0d9-95d9-43a6-b77f-bd41b3fd6383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" Nov 04 01:16:25.754415 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.754350 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26881d61-360e-43bb-83f8-3122e0026e41-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-ddd7x\" (UID: \"26881d61-360e-43bb-83f8-3122e0026e41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" Nov 04 01:16:25.754415 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.754366 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26881d61-360e-43bb-83f8-3122e0026e41-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-ddd7x\" (UID: \"26881d61-360e-43bb-83f8-3122e0026e41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" Nov 04 01:16:25.754674 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.754476 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26881d61-360e-43bb-83f8-3122e0026e41-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-ddd7x\" (UID: \"26881d61-360e-43bb-83f8-3122e0026e41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" Nov 04 01:16:25.754674 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.754512 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26881d61-360e-43bb-83f8-3122e0026e41-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-ddd7x\" (UID: \"26881d61-360e-43bb-83f8-3122e0026e41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" Nov 04 01:16:25.754674 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.754557 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26881d61-360e-43bb-83f8-3122e0026e41-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-ddd7x\" (UID: \"26881d61-360e-43bb-83f8-3122e0026e41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" Nov 04 01:16:25.754674 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.754619 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26881d61-360e-43bb-83f8-3122e0026e41-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-ddd7x\" (UID: \"26881d61-360e-43bb-83f8-3122e0026e41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" Nov 04 01:16:25.754674 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.754644 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26881d61-360e-43bb-83f8-3122e0026e41-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-ddd7x\" (UID: \"26881d61-360e-43bb-83f8-3122e0026e41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" Nov 04 01:16:25.754918 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.754691 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e65c0d9-95d9-43a6-b77f-bd41b3fd6383-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-bprgn\" (UID: \"5e65c0d9-95d9-43a6-b77f-bd41b3fd6383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" Nov 04 01:16:25.754918 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.754710 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e65c0d9-95d9-43a6-b77f-bd41b3fd6383-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-bprgn\" (UID: \"5e65c0d9-95d9-43a6-b77f-bd41b3fd6383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" Nov 04 01:16:25.754918 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.754734 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e65c0d9-95d9-43a6-b77f-bd41b3fd6383-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-bprgn\" (UID: \"5e65c0d9-95d9-43a6-b77f-bd41b3fd6383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" Nov 04 01:16:25.754918 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.754744 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26881d61-360e-43bb-83f8-3122e0026e41-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-ddd7x\" (UID: \"26881d61-360e-43bb-83f8-3122e0026e41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" Nov 04 01:16:25.754918 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.754762 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpmrz\" (UniqueName: \"kubernetes.io/projected/5e65c0d9-95d9-43a6-b77f-bd41b3fd6383-kube-api-access-bpmrz\") pod \"maas-default-gateway-openshift-default-75d7c785f-bprgn\" (UID: \"5e65c0d9-95d9-43a6-b77f-bd41b3fd6383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" Nov 04 01:16:25.754918 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.754791 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e65c0d9-95d9-43a6-b77f-bd41b3fd6383-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-bprgn\" (UID: \"5e65c0d9-95d9-43a6-b77f-bd41b3fd6383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" Nov 04 01:16:25.754918 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.754812 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e65c0d9-95d9-43a6-b77f-bd41b3fd6383-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-bprgn\" (UID: \"5e65c0d9-95d9-43a6-b77f-bd41b3fd6383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" Nov 04 01:16:25.754918 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.754811 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26881d61-360e-43bb-83f8-3122e0026e41-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-ddd7x\" (UID: \"26881d61-360e-43bb-83f8-3122e0026e41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" Nov 04 01:16:25.755328 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.754965 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26881d61-360e-43bb-83f8-3122e0026e41-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-ddd7x\" (UID: \"26881d61-360e-43bb-83f8-3122e0026e41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" Nov 04 01:16:25.755328 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.754991 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26881d61-360e-43bb-83f8-3122e0026e41-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-ddd7x\" (UID: \"26881d61-360e-43bb-83f8-3122e0026e41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" Nov 04 01:16:25.755328 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.755124 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e65c0d9-95d9-43a6-b77f-bd41b3fd6383-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-bprgn\" (UID: \"5e65c0d9-95d9-43a6-b77f-bd41b3fd6383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" Nov 04 01:16:25.755328 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.755156 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e65c0d9-95d9-43a6-b77f-bd41b3fd6383-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-bprgn\" (UID: \"5e65c0d9-95d9-43a6-b77f-bd41b3fd6383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" Nov 04 01:16:25.755328 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.755183 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e65c0d9-95d9-43a6-b77f-bd41b3fd6383-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-bprgn\" (UID: \"5e65c0d9-95d9-43a6-b77f-bd41b3fd6383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" Nov 04 01:16:25.755328 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.755314 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26881d61-360e-43bb-83f8-3122e0026e41-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-ddd7x\" (UID: \"26881d61-360e-43bb-83f8-3122e0026e41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" Nov 04 01:16:25.755530 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.755410 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e65c0d9-95d9-43a6-b77f-bd41b3fd6383-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-bprgn\" (UID: \"5e65c0d9-95d9-43a6-b77f-bd41b3fd6383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" Nov 04 01:16:25.755530 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.755434 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26881d61-360e-43bb-83f8-3122e0026e41-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-ddd7x\" (UID: \"26881d61-360e-43bb-83f8-3122e0026e41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" Nov 04 01:16:25.757053 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.757028 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26881d61-360e-43bb-83f8-3122e0026e41-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-ddd7x\" (UID: \"26881d61-360e-43bb-83f8-3122e0026e41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" Nov 04 01:16:25.757148 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.757092 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e65c0d9-95d9-43a6-b77f-bd41b3fd6383-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-bprgn\" (UID: \"5e65c0d9-95d9-43a6-b77f-bd41b3fd6383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" Nov 04 01:16:25.757414 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.757398 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26881d61-360e-43bb-83f8-3122e0026e41-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-ddd7x\" (UID: \"26881d61-360e-43bb-83f8-3122e0026e41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" Nov 04 01:16:25.762240 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.762221 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26881d61-360e-43bb-83f8-3122e0026e41-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-ddd7x\" (UID: \"26881d61-360e-43bb-83f8-3122e0026e41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" Nov 04 01:16:25.762700 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.762681 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmwth\" (UniqueName: \"kubernetes.io/projected/26881d61-360e-43bb-83f8-3122e0026e41-kube-api-access-qmwth\") pod \"maas-default-gateway-openshift-default-75d7c785f-ddd7x\" (UID: \"26881d61-360e-43bb-83f8-3122e0026e41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" Nov 04 01:16:25.856133 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.856074 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e65c0d9-95d9-43a6-b77f-bd41b3fd6383-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-bprgn\" (UID: \"5e65c0d9-95d9-43a6-b77f-bd41b3fd6383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" Nov 04 01:16:25.856133 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.856122 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpmrz\" (UniqueName: \"kubernetes.io/projected/5e65c0d9-95d9-43a6-b77f-bd41b3fd6383-kube-api-access-bpmrz\") pod \"maas-default-gateway-openshift-default-75d7c785f-bprgn\" (UID: \"5e65c0d9-95d9-43a6-b77f-bd41b3fd6383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" Nov 04 01:16:25.856303 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.856160 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e65c0d9-95d9-43a6-b77f-bd41b3fd6383-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-bprgn\" (UID: \"5e65c0d9-95d9-43a6-b77f-bd41b3fd6383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" Nov 04 01:16:25.856303 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.856184 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e65c0d9-95d9-43a6-b77f-bd41b3fd6383-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-bprgn\" (UID: \"5e65c0d9-95d9-43a6-b77f-bd41b3fd6383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" Nov 04 01:16:25.856768 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.856748 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e65c0d9-95d9-43a6-b77f-bd41b3fd6383-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-bprgn\" (UID: \"5e65c0d9-95d9-43a6-b77f-bd41b3fd6383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" Nov 04 01:16:25.858385 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.858367 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e65c0d9-95d9-43a6-b77f-bd41b3fd6383-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-bprgn\" (UID: \"5e65c0d9-95d9-43a6-b77f-bd41b3fd6383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" Nov 04 01:16:25.864357 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.864335 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpmrz\" (UniqueName: \"kubernetes.io/projected/5e65c0d9-95d9-43a6-b77f-bd41b3fd6383-kube-api-access-bpmrz\") pod \"maas-default-gateway-openshift-default-75d7c785f-bprgn\" (UID: \"5e65c0d9-95d9-43a6-b77f-bd41b3fd6383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" Nov 04 01:16:25.864432 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.864362 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e65c0d9-95d9-43a6-b77f-bd41b3fd6383-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-bprgn\" (UID: \"5e65c0d9-95d9-43a6-b77f-bd41b3fd6383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" Nov 04 01:16:25.917497 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.917480 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" Nov 04 01:16:25.954498 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:25.954474 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" Nov 04 01:16:26.197534 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:26.197508 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn"] Nov 04 01:16:26.199661 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:16:26.199635 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e65c0d9_95d9_43a6_b77f_bd41b3fd6383.slice/crio-ab3a3974773ee6edfca879bd1c89da3e93b5312a2f6773df161f1254ee24555f WatchSource:0}: Error finding container ab3a3974773ee6edfca879bd1c89da3e93b5312a2f6773df161f1254ee24555f: Status 404 returned error can't find the container with id ab3a3974773ee6edfca879bd1c89da3e93b5312a2f6773df161f1254ee24555f Nov 04 01:16:26.202181 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:26.202148 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:16:26.202249 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:26.202211 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:16:26.202249 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:26.202242 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:16:26.238423 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:26.238399 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x"] Nov 04 01:16:26.240845 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:16:26.240817 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26881d61_360e_43bb_83f8_3122e0026e41.slice/crio-29dcf9fcc8bfa410ad09ed3b6a8c1ee0f7bbd740b65b71c4c7a5939d7cf52591 WatchSource:0}: Error finding container 29dcf9fcc8bfa410ad09ed3b6a8c1ee0f7bbd740b65b71c4c7a5939d7cf52591: Status 404 returned error can't find the container with id 29dcf9fcc8bfa410ad09ed3b6a8c1ee0f7bbd740b65b71c4c7a5939d7cf52591 Nov 04 01:16:26.243100 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:26.243066 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:16:26.243196 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:26.243132 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:16:26.243196 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:26.243163 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:16:26.522487 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:26.522404 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" event={"ID":"26881d61-360e-43bb-83f8-3122e0026e41","Type":"ContainerStarted","Data":"8ebd40b43b547234ac7ab62846c653fddad1317716425e1720f88d4ac06b3b03"} Nov 04 01:16:26.522487 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:26.522471 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" event={"ID":"26881d61-360e-43bb-83f8-3122e0026e41","Type":"ContainerStarted","Data":"29dcf9fcc8bfa410ad09ed3b6a8c1ee0f7bbd740b65b71c4c7a5939d7cf52591"} Nov 04 01:16:26.524263 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:26.524237 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" event={"ID":"5e65c0d9-95d9-43a6-b77f-bd41b3fd6383","Type":"ContainerStarted","Data":"5cb264f89e16aa0a3ce16a97ab6e63933c756e679d435767a682573f7649cf58"} Nov 04 01:16:26.524398 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:26.524276 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" event={"ID":"5e65c0d9-95d9-43a6-b77f-bd41b3fd6383","Type":"ContainerStarted","Data":"ab3a3974773ee6edfca879bd1c89da3e93b5312a2f6773df161f1254ee24555f"} Nov 04 01:16:26.545566 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:26.545513 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" podStartSLOduration=1.5454996749999999 podStartE2EDuration="1.545499675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:26.543416162 +0000 UTC m=+1071.485265100" watchObservedRunningTime="2026-11-04 01:16:26.545499675 +0000 UTC m=+1071.487348577" Nov 04 01:16:26.564603 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:26.564537 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" podStartSLOduration=1.5645195840000001 podStartE2EDuration="1.564519584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:26.564002926 +0000 UTC m=+1071.505851856" watchObservedRunningTime="2026-11-04 01:16:26.564519584 +0000 UTC m=+1071.506368489" Nov 04 01:16:26.918068 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:26.917966 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" Nov 04 01:16:26.957949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:26.957023 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" Nov 04 01:16:27.918705 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:27.918650 7059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bprgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 04 01:16:27.919105 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:27.918716 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" podUID="5e65c0d9-95d9-43a6-b77f-bd41b3fd6383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 04 01:16:27.965164 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:27.965131 7059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-ddd7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 04 01:16:27.965392 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:27.965361 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" podUID="26881d61-360e-43bb-83f8-3122e0026e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 04 01:16:28.918099 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:28.918057 7059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bprgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 04 01:16:28.918315 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:28.918119 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" podUID="5e65c0d9-95d9-43a6-b77f-bd41b3fd6383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 04 01:16:28.959958 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:28.959342 7059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-ddd7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 04 01:16:28.959958 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:28.959402 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" podUID="26881d61-360e-43bb-83f8-3122e0026e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 04 01:16:29.918385 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:29.918350 7059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bprgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 04 01:16:29.918555 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:29.918426 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" podUID="5e65c0d9-95d9-43a6-b77f-bd41b3fd6383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 04 01:16:29.955346 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:29.955310 7059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-ddd7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 04 01:16:29.955595 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:29.955566 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" podUID="26881d61-360e-43bb-83f8-3122e0026e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 04 01:16:30.918215 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:30.918178 7059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bprgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 04 01:16:30.918601 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:30.918245 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" podUID="5e65c0d9-95d9-43a6-b77f-bd41b3fd6383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 04 01:16:30.959950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:30.956274 7059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-ddd7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 04 01:16:30.959950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:30.956346 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" podUID="26881d61-360e-43bb-83f8-3122e0026e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 04 01:16:31.918730 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:31.918689 7059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bprgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 04 01:16:31.919142 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:31.918752 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" podUID="5e65c0d9-95d9-43a6-b77f-bd41b3fd6383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 04 01:16:31.956613 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:31.956580 7059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-ddd7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 04 01:16:31.956908 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:31.956863 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" podUID="26881d61-360e-43bb-83f8-3122e0026e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 04 01:16:33.185175 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:33.185144 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" Nov 04 01:16:33.185804 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:33.185786 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" Nov 04 01:16:33.186803 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:33.186774 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" Nov 04 01:16:33.186917 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:33.186825 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" Nov 04 01:16:33.188493 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:33.188457 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" Nov 04 01:16:33.193222 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:33.193198 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" Nov 04 01:16:33.442717 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:33.442629 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7"] Nov 04 01:16:33.448087 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:33.448063 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7" Nov 04 01:16:33.454971 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:33.454945 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-njzcd\"" Nov 04 01:16:33.455097 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:33.455078 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 04 01:16:33.455170 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:33.455144 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7"] Nov 04 01:16:33.472455 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:33.472434 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 04 01:16:33.472607 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:33.472492 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 04 01:16:33.567526 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:33.567097 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23ea0f8f-58d8-43aa-b25b-f09ea8321e21-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7\" (UID: \"23ea0f8f-58d8-43aa-b25b-f09ea8321e21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7" Nov 04 01:16:33.567526 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:33.567187 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23ea0f8f-58d8-43aa-b25b-f09ea8321e21-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7\" (UID: \"23ea0f8f-58d8-43aa-b25b-f09ea8321e21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7" Nov 04 01:16:33.567526 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:33.567222 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23ea0f8f-58d8-43aa-b25b-f09ea8321e21-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7\" (UID: \"23ea0f8f-58d8-43aa-b25b-f09ea8321e21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7" Nov 04 01:16:33.567526 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:33.567260 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23ea0f8f-58d8-43aa-b25b-f09ea8321e21-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7\" (UID: \"23ea0f8f-58d8-43aa-b25b-f09ea8321e21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7" Nov 04 01:16:33.567526 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:33.567298 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23ea0f8f-58d8-43aa-b25b-f09ea8321e21-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7\" (UID: \"23ea0f8f-58d8-43aa-b25b-f09ea8321e21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7" Nov 04 01:16:33.567526 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:33.567341 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4f74\" (UniqueName: \"kubernetes.io/projected/23ea0f8f-58d8-43aa-b25b-f09ea8321e21-kube-api-access-m4f74\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7\" (UID: \"23ea0f8f-58d8-43aa-b25b-f09ea8321e21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7" Nov 04 01:16:33.668153 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:33.668119 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23ea0f8f-58d8-43aa-b25b-f09ea8321e21-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7\" (UID: \"23ea0f8f-58d8-43aa-b25b-f09ea8321e21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7" Nov 04 01:16:33.668322 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:33.668167 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23ea0f8f-58d8-43aa-b25b-f09ea8321e21-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7\" (UID: \"23ea0f8f-58d8-43aa-b25b-f09ea8321e21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7" Nov 04 01:16:33.668382 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:33.668329 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4f74\" (UniqueName: \"kubernetes.io/projected/23ea0f8f-58d8-43aa-b25b-f09ea8321e21-kube-api-access-m4f74\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7\" (UID: \"23ea0f8f-58d8-43aa-b25b-f09ea8321e21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7" Nov 04 01:16:33.668460 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:33.668445 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23ea0f8f-58d8-43aa-b25b-f09ea8321e21-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7\" (UID: \"23ea0f8f-58d8-43aa-b25b-f09ea8321e21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7" Nov 04 01:16:33.668599 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:33.668564 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23ea0f8f-58d8-43aa-b25b-f09ea8321e21-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7\" (UID: \"23ea0f8f-58d8-43aa-b25b-f09ea8321e21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7" Nov 04 01:16:33.668739 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:33.668656 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23ea0f8f-58d8-43aa-b25b-f09ea8321e21-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7\" (UID: \"23ea0f8f-58d8-43aa-b25b-f09ea8321e21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7" Nov 04 01:16:33.668739 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:33.668692 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23ea0f8f-58d8-43aa-b25b-f09ea8321e21-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7\" (UID: \"23ea0f8f-58d8-43aa-b25b-f09ea8321e21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7" Nov 04 01:16:33.668918 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:33.668895 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23ea0f8f-58d8-43aa-b25b-f09ea8321e21-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7\" (UID: \"23ea0f8f-58d8-43aa-b25b-f09ea8321e21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7" Nov 04 01:16:33.669098 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:33.669078 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23ea0f8f-58d8-43aa-b25b-f09ea8321e21-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7\" (UID: \"23ea0f8f-58d8-43aa-b25b-f09ea8321e21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7" Nov 04 01:16:33.670597 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:33.670579 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23ea0f8f-58d8-43aa-b25b-f09ea8321e21-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7\" (UID: \"23ea0f8f-58d8-43aa-b25b-f09ea8321e21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7" Nov 04 01:16:33.671264 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:33.671247 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23ea0f8f-58d8-43aa-b25b-f09ea8321e21-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7\" (UID: \"23ea0f8f-58d8-43aa-b25b-f09ea8321e21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7" Nov 04 01:16:33.677162 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:33.677139 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4f74\" (UniqueName: \"kubernetes.io/projected/23ea0f8f-58d8-43aa-b25b-f09ea8321e21-kube-api-access-m4f74\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7\" (UID: \"23ea0f8f-58d8-43aa-b25b-f09ea8321e21\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7" Nov 04 01:16:33.767975 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:33.767891 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7" Nov 04 01:16:34.046087 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:34.046058 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7"] Nov 04 01:16:34.047752 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:16:34.047718 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23ea0f8f_58d8_43aa_b25b_f09ea8321e21.slice/crio-8f140b28d23a26bd34fe32d1c0c5b1b830b88ed98500e328025b1e0456127fdd WatchSource:0}: Error finding container 8f140b28d23a26bd34fe32d1c0c5b1b830b88ed98500e328025b1e0456127fdd: Status 404 returned error can't find the container with id 8f140b28d23a26bd34fe32d1c0c5b1b830b88ed98500e328025b1e0456127fdd Nov 04 01:16:34.614590 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:34.614553 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7" event={"ID":"23ea0f8f-58d8-43aa-b25b-f09ea8321e21","Type":"ContainerStarted","Data":"8f140b28d23a26bd34fe32d1c0c5b1b830b88ed98500e328025b1e0456127fdd"} Nov 04 01:16:36.597949 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:16:36.584230 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:16:36.635644 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:36.634966 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7" event={"ID":"23ea0f8f-58d8-43aa-b25b-f09ea8321e21","Type":"ContainerStarted","Data":"05f0ae0724c29e39673a4b1888ee3df64be1a83c280b8b8b8ba3bdd985876881"} Nov 04 01:16:36.702948 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:36.687975 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7" podStartSLOduration=1.5865867150000001 podStartE2EDuration="3.687955162s" totalImagesPullingTime="2.101368447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:36.685352738 +0000 UTC m=+1081.627201641" watchObservedRunningTime="2026-11-04 01:16:36.687955162 +0000 UTC m=+1081.629804065" Nov 04 01:16:36.769948 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:36.769009 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7" Nov 04 01:16:39.175950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:39.171260 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm"] Nov 04 01:16:39.226270 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:39.226226 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:16:39.232431 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:39.232407 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 04 01:16:39.319066 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:39.318969 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abeae128-5b8c-4d5e-9592-f64e9c6bd4b5-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm\" (UID: \"abeae128-5b8c-4d5e-9592-f64e9c6bd4b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:16:39.319765 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:39.319738 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abeae128-5b8c-4d5e-9592-f64e9c6bd4b5-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm\" (UID: \"abeae128-5b8c-4d5e-9592-f64e9c6bd4b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:16:39.322835 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:39.322808 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppchq\" (UniqueName: \"kubernetes.io/projected/abeae128-5b8c-4d5e-9592-f64e9c6bd4b5-kube-api-access-ppchq\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm\" (UID: \"abeae128-5b8c-4d5e-9592-f64e9c6bd4b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:16:39.323023 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:39.323004 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abeae128-5b8c-4d5e-9592-f64e9c6bd4b5-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm\" (UID: \"abeae128-5b8c-4d5e-9592-f64e9c6bd4b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:16:39.323247 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:39.323227 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abeae128-5b8c-4d5e-9592-f64e9c6bd4b5-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm\" (UID: \"abeae128-5b8c-4d5e-9592-f64e9c6bd4b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:16:39.324655 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:39.324631 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abeae128-5b8c-4d5e-9592-f64e9c6bd4b5-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm\" (UID: \"abeae128-5b8c-4d5e-9592-f64e9c6bd4b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:16:39.326721 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:39.326560 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm"] Nov 04 01:16:39.425855 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:39.425824 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppchq\" (UniqueName: \"kubernetes.io/projected/abeae128-5b8c-4d5e-9592-f64e9c6bd4b5-kube-api-access-ppchq\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm\" (UID: \"abeae128-5b8c-4d5e-9592-f64e9c6bd4b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:16:39.426355 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:39.426296 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abeae128-5b8c-4d5e-9592-f64e9c6bd4b5-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm\" (UID: \"abeae128-5b8c-4d5e-9592-f64e9c6bd4b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:16:39.426566 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:39.426553 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abeae128-5b8c-4d5e-9592-f64e9c6bd4b5-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm\" (UID: \"abeae128-5b8c-4d5e-9592-f64e9c6bd4b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:16:39.426991 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:39.426976 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abeae128-5b8c-4d5e-9592-f64e9c6bd4b5-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm\" (UID: \"abeae128-5b8c-4d5e-9592-f64e9c6bd4b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:16:39.427168 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:39.427151 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abeae128-5b8c-4d5e-9592-f64e9c6bd4b5-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm\" (UID: \"abeae128-5b8c-4d5e-9592-f64e9c6bd4b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:16:39.427332 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:39.427314 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abeae128-5b8c-4d5e-9592-f64e9c6bd4b5-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm\" (UID: \"abeae128-5b8c-4d5e-9592-f64e9c6bd4b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:16:39.427437 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:39.427419 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abeae128-5b8c-4d5e-9592-f64e9c6bd4b5-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm\" (UID: \"abeae128-5b8c-4d5e-9592-f64e9c6bd4b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:16:39.427562 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:39.427392 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abeae128-5b8c-4d5e-9592-f64e9c6bd4b5-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm\" (UID: \"abeae128-5b8c-4d5e-9592-f64e9c6bd4b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:16:39.427623 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:39.427543 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abeae128-5b8c-4d5e-9592-f64e9c6bd4b5-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm\" (UID: \"abeae128-5b8c-4d5e-9592-f64e9c6bd4b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:16:39.429089 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:39.429058 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abeae128-5b8c-4d5e-9592-f64e9c6bd4b5-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm\" (UID: \"abeae128-5b8c-4d5e-9592-f64e9c6bd4b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:16:39.429267 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:39.429251 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abeae128-5b8c-4d5e-9592-f64e9c6bd4b5-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm\" (UID: \"abeae128-5b8c-4d5e-9592-f64e9c6bd4b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:16:39.433878 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:39.433854 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppchq\" (UniqueName: \"kubernetes.io/projected/abeae128-5b8c-4d5e-9592-f64e9c6bd4b5-kube-api-access-ppchq\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm\" (UID: \"abeae128-5b8c-4d5e-9592-f64e9c6bd4b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:16:39.617766 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:39.617730 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:16:39.893572 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:39.893547 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm"] Nov 04 01:16:39.895572 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:16:39.895534 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabeae128_5b8c_4d5e_9592_f64e9c6bd4b5.slice/crio-d750fdb0a0302d4225c324bc6870025343e9b9754d6d4bb0cd8d3c84b401b9dd WatchSource:0}: Error finding container d750fdb0a0302d4225c324bc6870025343e9b9754d6d4bb0cd8d3c84b401b9dd: Status 404 returned error can't find the container with id d750fdb0a0302d4225c324bc6870025343e9b9754d6d4bb0cd8d3c84b401b9dd Nov 04 01:16:40.709903 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:40.709868 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" event={"ID":"abeae128-5b8c-4d5e-9592-f64e9c6bd4b5","Type":"ContainerStarted","Data":"f172aee9c971db6b874e5ec28d6b677592f4de12fedfc212e7e208751fa272b0"} Nov 04 01:16:40.710448 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:40.710425 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" event={"ID":"abeae128-5b8c-4d5e-9592-f64e9c6bd4b5","Type":"ContainerStarted","Data":"d750fdb0a0302d4225c324bc6870025343e9b9754d6d4bb0cd8d3c84b401b9dd"} Nov 04 01:16:40.769947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:40.768269 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podStartSLOduration=1.481381692 podStartE2EDuration="1.768249358s" totalImagesPullingTime="286.867666ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:40.764882639 +0000 UTC m=+1085.706731543" watchObservedRunningTime="2026-11-04 01:16:40.768249358 +0000 UTC m=+1085.710098262" Nov 04 01:16:41.618878 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:41.618846 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:16:41.714949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:41.712498 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-tl4kv"] Nov 04 01:16:41.727121 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:41.727096 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-tl4kv"] Nov 04 01:16:41.727385 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:41.727369 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tl4kv" Nov 04 01:16:41.732895 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:41.732836 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 04 01:16:41.791335 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:41.790497 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tl4kv\" (UID: \"c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tl4kv" Nov 04 01:16:41.791335 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:41.790551 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tl4kv\" (UID: \"c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tl4kv" Nov 04 01:16:41.791335 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:41.790583 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tl4kv\" (UID: \"c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tl4kv" Nov 04 01:16:41.791335 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:41.790629 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tl4kv\" (UID: \"c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tl4kv" Nov 04 01:16:41.791335 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:41.790722 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwbmv\" (UniqueName: \"kubernetes.io/projected/c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd-kube-api-access-nwbmv\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tl4kv\" (UID: \"c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tl4kv" Nov 04 01:16:41.791335 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:41.790777 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tl4kv\" (UID: \"c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tl4kv" Nov 04 01:16:41.920406 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:41.918079 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwbmv\" (UniqueName: \"kubernetes.io/projected/c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd-kube-api-access-nwbmv\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tl4kv\" (UID: \"c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tl4kv" Nov 04 01:16:41.920406 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:41.918160 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tl4kv\" (UID: \"c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tl4kv" Nov 04 01:16:41.920406 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:41.918260 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tl4kv\" (UID: \"c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tl4kv" Nov 04 01:16:41.920406 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:41.918289 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tl4kv\" (UID: \"c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tl4kv" Nov 04 01:16:41.920406 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:41.918319 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tl4kv\" (UID: \"c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tl4kv" Nov 04 01:16:41.920406 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:41.918365 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tl4kv\" (UID: \"c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tl4kv" Nov 04 01:16:41.925985 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:41.925125 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tl4kv\" (UID: \"c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tl4kv" Nov 04 01:16:41.925985 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:41.925387 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tl4kv\" (UID: \"c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tl4kv" Nov 04 01:16:41.925985 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:41.925623 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tl4kv\" (UID: \"c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tl4kv" Nov 04 01:16:41.926424 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:41.926398 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tl4kv\" (UID: \"c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tl4kv" Nov 04 01:16:41.928365 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:41.928343 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tl4kv\" (UID: \"c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tl4kv" Nov 04 01:16:41.928505 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:41.928440 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwbmv\" (UniqueName: \"kubernetes.io/projected/c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd-kube-api-access-nwbmv\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-tl4kv\" (UID: \"c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tl4kv" Nov 04 01:16:42.106194 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:42.106161 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tl4kv" Nov 04 01:16:42.432632 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:42.432608 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-tl4kv"] Nov 04 01:16:42.434137 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:16:42.434102 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc973cb5b_66ec_4ec2_9cb3_0b8faa77f6bd.slice/crio-02bbd8a999d52a30833384533afb910e47202ec066469b40e83c082a123534b4 WatchSource:0}: Error finding container 02bbd8a999d52a30833384533afb910e47202ec066469b40e83c082a123534b4: Status 404 returned error can't find the container with id 02bbd8a999d52a30833384533afb910e47202ec066469b40e83c082a123534b4 Nov 04 01:16:42.558911 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:42.558844 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-9clfp"] Nov 04 01:16:42.570687 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:42.570661 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9clfp" Nov 04 01:16:42.572841 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:42.572816 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-9clfp"] Nov 04 01:16:42.579847 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:42.579827 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 04 01:16:42.731017 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:42.730994 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e6a52c4-c655-43d6-8a0e-200e8523e315-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9clfp\" (UID: \"7e6a52c4-c655-43d6-8a0e-200e8523e315\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9clfp" Nov 04 01:16:42.731422 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:42.731031 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e6a52c4-c655-43d6-8a0e-200e8523e315-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9clfp\" (UID: \"7e6a52c4-c655-43d6-8a0e-200e8523e315\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9clfp" Nov 04 01:16:42.731422 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:42.731104 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e6a52c4-c655-43d6-8a0e-200e8523e315-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9clfp\" (UID: \"7e6a52c4-c655-43d6-8a0e-200e8523e315\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9clfp" Nov 04 01:16:42.731422 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:42.731220 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e6a52c4-c655-43d6-8a0e-200e8523e315-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9clfp\" (UID: \"7e6a52c4-c655-43d6-8a0e-200e8523e315\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9clfp" Nov 04 01:16:42.731422 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:42.731259 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqbpb\" (UniqueName: \"kubernetes.io/projected/7e6a52c4-c655-43d6-8a0e-200e8523e315-kube-api-access-pqbpb\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9clfp\" (UID: \"7e6a52c4-c655-43d6-8a0e-200e8523e315\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9clfp" Nov 04 01:16:42.731422 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:42.731325 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e6a52c4-c655-43d6-8a0e-200e8523e315-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9clfp\" (UID: \"7e6a52c4-c655-43d6-8a0e-200e8523e315\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9clfp" Nov 04 01:16:42.731686 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:42.731476 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tl4kv" event={"ID":"c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd","Type":"ContainerStarted","Data":"02bbd8a999d52a30833384533afb910e47202ec066469b40e83c082a123534b4"} Nov 04 01:16:42.832898 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:42.832840 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e6a52c4-c655-43d6-8a0e-200e8523e315-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9clfp\" (UID: \"7e6a52c4-c655-43d6-8a0e-200e8523e315\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9clfp" Nov 04 01:16:42.832898 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:42.832881 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e6a52c4-c655-43d6-8a0e-200e8523e315-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9clfp\" (UID: \"7e6a52c4-c655-43d6-8a0e-200e8523e315\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9clfp" Nov 04 01:16:42.833060 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:42.832910 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e6a52c4-c655-43d6-8a0e-200e8523e315-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9clfp\" (UID: \"7e6a52c4-c655-43d6-8a0e-200e8523e315\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9clfp" Nov 04 01:16:42.833060 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:42.832960 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e6a52c4-c655-43d6-8a0e-200e8523e315-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9clfp\" (UID: \"7e6a52c4-c655-43d6-8a0e-200e8523e315\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9clfp" Nov 04 01:16:42.833060 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:42.832995 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqbpb\" (UniqueName: \"kubernetes.io/projected/7e6a52c4-c655-43d6-8a0e-200e8523e315-kube-api-access-pqbpb\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9clfp\" (UID: \"7e6a52c4-c655-43d6-8a0e-200e8523e315\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9clfp" Nov 04 01:16:42.833060 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:42.833056 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e6a52c4-c655-43d6-8a0e-200e8523e315-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9clfp\" (UID: \"7e6a52c4-c655-43d6-8a0e-200e8523e315\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9clfp" Nov 04 01:16:42.833280 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:42.833234 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e6a52c4-c655-43d6-8a0e-200e8523e315-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9clfp\" (UID: \"7e6a52c4-c655-43d6-8a0e-200e8523e315\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9clfp" Nov 04 01:16:42.833348 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:42.833280 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e6a52c4-c655-43d6-8a0e-200e8523e315-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9clfp\" (UID: \"7e6a52c4-c655-43d6-8a0e-200e8523e315\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9clfp" Nov 04 01:16:42.833411 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:42.833361 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e6a52c4-c655-43d6-8a0e-200e8523e315-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9clfp\" (UID: \"7e6a52c4-c655-43d6-8a0e-200e8523e315\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9clfp" Nov 04 01:16:42.835228 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:42.835211 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e6a52c4-c655-43d6-8a0e-200e8523e315-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9clfp\" (UID: \"7e6a52c4-c655-43d6-8a0e-200e8523e315\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9clfp" Nov 04 01:16:42.835412 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:42.835395 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e6a52c4-c655-43d6-8a0e-200e8523e315-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9clfp\" (UID: \"7e6a52c4-c655-43d6-8a0e-200e8523e315\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9clfp" Nov 04 01:16:42.841207 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:42.841188 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqbpb\" (UniqueName: \"kubernetes.io/projected/7e6a52c4-c655-43d6-8a0e-200e8523e315-kube-api-access-pqbpb\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-9clfp\" (UID: \"7e6a52c4-c655-43d6-8a0e-200e8523e315\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9clfp" Nov 04 01:16:42.886447 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:42.886418 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9clfp" Nov 04 01:16:43.180454 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:43.180423 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-9clfp"] Nov 04 01:16:43.181412 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:16:43.181378 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e6a52c4_c655_43d6_8a0e_200e8523e315.slice/crio-223056eab62a35d14ac68f136a1d8072a0f6c2f50fc3ac7e3548f19b53fb0807 WatchSource:0}: Error finding container 223056eab62a35d14ac68f136a1d8072a0f6c2f50fc3ac7e3548f19b53fb0807: Status 404 returned error can't find the container with id 223056eab62a35d14ac68f136a1d8072a0f6c2f50fc3ac7e3548f19b53fb0807 Nov 04 01:16:43.740376 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:43.740249 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tl4kv" event={"ID":"c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd","Type":"ContainerStarted","Data":"6e07561c2816cabd7e530f0e4bbc6efbe16eafe9530db5fce81b91675195800d"} Nov 04 01:16:43.742306 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:43.742281 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9clfp" event={"ID":"7e6a52c4-c655-43d6-8a0e-200e8523e315","Type":"ContainerStarted","Data":"315f80de03928af3d360e37bd77c779e3e3bf31897e08b991708a3680bbf88f0"} Nov 04 01:16:43.742443 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:43.742328 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9clfp" event={"ID":"7e6a52c4-c655-43d6-8a0e-200e8523e315","Type":"ContainerStarted","Data":"223056eab62a35d14ac68f136a1d8072a0f6c2f50fc3ac7e3548f19b53fb0807"} Nov 04 01:16:43.759354 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:43.759299 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tl4kv" podStartSLOduration=2.466975773 podStartE2EDuration="2.75928171s" totalImagesPullingTime="292.305937ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:43.757492109 +0000 UTC m=+1088.699341014" watchObservedRunningTime="2026-11-04 01:16:43.75928171 +0000 UTC m=+1088.701130615" Nov 04 01:16:43.768275 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:43.768248 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7" Nov 04 01:16:43.777671 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:43.777622 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9clfp" podStartSLOduration=1.475498916 podStartE2EDuration="1.777608061s" totalImagesPullingTime="302.109145ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:43.775248423 +0000 UTC m=+1088.717097328" watchObservedRunningTime="2026-11-04 01:16:43.777608061 +0000 UTC m=+1088.719456966" Nov 04 01:16:43.792063 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:43.792039 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7" Nov 04 01:16:43.887949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:43.887006 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9clfp" Nov 04 01:16:44.107459 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:44.107366 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tl4kv" Nov 04 01:16:44.808039 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:44.795415 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7" Nov 04 01:16:48.957088 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:48.957048 7059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-7rw6b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:16:48.957828 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:48.957775 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" podUID="994ce0d7-9fe2-494e-a915-94fe06da822e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:49.230166 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:49.230064 7059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bprgn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:16:49.230166 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:49.230135 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" podUID="5e65c0d9-95d9-43a6-b77f-bd41b3fd6383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:49.231049 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:49.231023 7059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-ddd7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:16:49.231204 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:49.231065 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" podUID="26881d61-360e-43bb-83f8-3122e0026e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:49.618222 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:49.618153 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:16:49.630766 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:49.630745 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:16:49.840840 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:49.840818 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:16:52.107406 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:52.107369 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tl4kv" Nov 04 01:16:52.120048 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:52.120013 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tl4kv" Nov 04 01:16:52.856842 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:52.856809 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-tl4kv" Nov 04 01:16:52.887080 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:52.887051 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9clfp" Nov 04 01:16:52.905145 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:52.905119 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9clfp" Nov 04 01:16:53.861912 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:53.861883 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-9clfp" Nov 04 01:16:57.916208 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:57.916175 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-648b7c5784-dn7sb"] Nov 04 01:16:57.916644 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:57.916374 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-648b7c5784-dn7sb" podUID="bd2407fe-d5a7-4207-8e0a-1bc1980abeb0" containerName="authorino" containerID="cri-o://eb4939e7b4f6b726ebc48a0c3fd06c2b38fb61f2ed5d4db9a2a2979d6a9faa64" gracePeriod=30 Nov 04 01:16:58.165379 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:58.165354 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-648b7c5784-dn7sb" Nov 04 01:16:58.290053 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:58.290022 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd2407fe-d5a7-4207-8e0a-1bc1980abeb0-tls-cert\" (UniqueName: \"kubernetes.io/secret/bd2407fe-d5a7-4207-8e0a-1bc1980abeb0-tls-cert\") pod \"bd2407fe-d5a7-4207-8e0a-1bc1980abeb0\" (UID: \"bd2407fe-d5a7-4207-8e0a-1bc1980abeb0\") " Nov 04 01:16:58.290254 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:58.290091 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd2407fe-d5a7-4207-8e0a-1bc1980abeb0-kube-api-access-l24xd\" (UniqueName: \"kubernetes.io/projected/bd2407fe-d5a7-4207-8e0a-1bc1980abeb0-kube-api-access-l24xd\") pod \"bd2407fe-d5a7-4207-8e0a-1bc1980abeb0\" (UID: \"bd2407fe-d5a7-4207-8e0a-1bc1980abeb0\") " Nov 04 01:16:58.292269 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:58.292242 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd2407fe-d5a7-4207-8e0a-1bc1980abeb0-kube-api-access-l24xd" pod "bd2407fe-d5a7-4207-8e0a-1bc1980abeb0" (UID: "bd2407fe-d5a7-4207-8e0a-1bc1980abeb0"). InnerVolumeSpecName "kube-api-access-l24xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:16:58.301070 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:58.301043 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd2407fe-d5a7-4207-8e0a-1bc1980abeb0-tls-cert" pod "bd2407fe-d5a7-4207-8e0a-1bc1980abeb0" (UID: "bd2407fe-d5a7-4207-8e0a-1bc1980abeb0"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:16:58.391750 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:58.391717 7059 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bd2407fe-d5a7-4207-8e0a-1bc1980abeb0-tls-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:16:58.391750 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:58.391744 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l24xd\" (UniqueName: \"kubernetes.io/projected/bd2407fe-d5a7-4207-8e0a-1bc1980abeb0-kube-api-access-l24xd\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:16:58.489056 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:58.489029 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-vvm6j"] Nov 04 01:16:58.489454 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:58.489443 7059 cpu_manager.go:414] "containerMap: removing container" podUID="bd2407fe-d5a7-4207-8e0a-1bc1980abeb0" containerName="authorino" Nov 04 01:16:58.489506 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:58.489456 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd2407fe-d5a7-4207-8e0a-1bc1980abeb0" containerName="authorino" Nov 04 01:16:58.489541 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:58.489530 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd2407fe-d5a7-4207-8e0a-1bc1980abeb0" containerName="authorino" Nov 04 01:16:58.493164 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:58.493147 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-vvm6j" Nov 04 01:16:58.498497 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:58.498407 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 04 01:16:58.499807 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:58.499760 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-5zkl7\"" Nov 04 01:16:58.499807 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:58.499761 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 04 01:16:58.502466 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:58.502445 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-vvm6j"] Nov 04 01:16:58.593758 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:58.593686 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e927034-cbc7-47c7-919f-a0254565ac99-metrics-certs\") pod \"ai-gateway-controller-7cb8f6bb64-vvm6j\" (UID: \"5e927034-cbc7-47c7-919f-a0254565ac99\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-vvm6j" Nov 04 01:16:58.593758 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:58.593721 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkhgn\" (UniqueName: \"kubernetes.io/projected/5e927034-cbc7-47c7-919f-a0254565ac99-kube-api-access-rkhgn\") pod \"ai-gateway-controller-7cb8f6bb64-vvm6j\" (UID: \"5e927034-cbc7-47c7-919f-a0254565ac99\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-vvm6j" Nov 04 01:16:58.694642 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:58.694611 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e927034-cbc7-47c7-919f-a0254565ac99-metrics-certs\") pod \"ai-gateway-controller-7cb8f6bb64-vvm6j\" (UID: \"5e927034-cbc7-47c7-919f-a0254565ac99\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-vvm6j" Nov 04 01:16:58.694808 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:58.694645 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkhgn\" (UniqueName: \"kubernetes.io/projected/5e927034-cbc7-47c7-919f-a0254565ac99-kube-api-access-rkhgn\") pod \"ai-gateway-controller-7cb8f6bb64-vvm6j\" (UID: \"5e927034-cbc7-47c7-919f-a0254565ac99\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-vvm6j" Nov 04 01:16:58.697219 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:58.697195 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e927034-cbc7-47c7-919f-a0254565ac99-metrics-certs\") pod \"ai-gateway-controller-7cb8f6bb64-vvm6j\" (UID: \"5e927034-cbc7-47c7-919f-a0254565ac99\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-vvm6j" Nov 04 01:16:58.702570 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:58.702551 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkhgn\" (UniqueName: \"kubernetes.io/projected/5e927034-cbc7-47c7-919f-a0254565ac99-kube-api-access-rkhgn\") pod \"ai-gateway-controller-7cb8f6bb64-vvm6j\" (UID: \"5e927034-cbc7-47c7-919f-a0254565ac99\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-vvm6j" Nov 04 01:16:58.809914 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:58.809872 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-vvm6j" Nov 04 01:16:58.899943 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:58.895076 7059 generic.go:356] "Generic (PLEG): container finished" podID="bd2407fe-d5a7-4207-8e0a-1bc1980abeb0" containerID="eb4939e7b4f6b726ebc48a0c3fd06c2b38fb61f2ed5d4db9a2a2979d6a9faa64" exitCode=0 Nov 04 01:16:58.899943 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:58.895129 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-648b7c5784-dn7sb" event={"ID":"bd2407fe-d5a7-4207-8e0a-1bc1980abeb0","Type":"ContainerDied","Data":"eb4939e7b4f6b726ebc48a0c3fd06c2b38fb61f2ed5d4db9a2a2979d6a9faa64"} Nov 04 01:16:58.899943 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:58.895165 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-648b7c5784-dn7sb" event={"ID":"bd2407fe-d5a7-4207-8e0a-1bc1980abeb0","Type":"ContainerDied","Data":"e048003e48a6fe2a4a4b9973ca7414949909442a956a5b087072f71cdaec5f6d"} Nov 04 01:16:58.899943 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:58.895185 7059 scope.go:122] "RemoveContainer" containerID="eb4939e7b4f6b726ebc48a0c3fd06c2b38fb61f2ed5d4db9a2a2979d6a9faa64" Nov 04 01:16:58.899943 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:58.895328 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-648b7c5784-dn7sb" Nov 04 01:16:58.927222 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:58.926409 7059 scope.go:122] "RemoveContainer" containerID="eb4939e7b4f6b726ebc48a0c3fd06c2b38fb61f2ed5d4db9a2a2979d6a9faa64" Nov 04 01:16:58.927222 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:16:58.926766 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb4939e7b4f6b726ebc48a0c3fd06c2b38fb61f2ed5d4db9a2a2979d6a9faa64\": container with ID starting with eb4939e7b4f6b726ebc48a0c3fd06c2b38fb61f2ed5d4db9a2a2979d6a9faa64 not found: ID does not exist" containerID="eb4939e7b4f6b726ebc48a0c3fd06c2b38fb61f2ed5d4db9a2a2979d6a9faa64" Nov 04 01:16:58.927222 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:58.926817 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb4939e7b4f6b726ebc48a0c3fd06c2b38fb61f2ed5d4db9a2a2979d6a9faa64"} err="failed to get container status \"eb4939e7b4f6b726ebc48a0c3fd06c2b38fb61f2ed5d4db9a2a2979d6a9faa64\": rpc error: code = NotFound desc = could not find container \"eb4939e7b4f6b726ebc48a0c3fd06c2b38fb61f2ed5d4db9a2a2979d6a9faa64\": container with ID starting with eb4939e7b4f6b726ebc48a0c3fd06c2b38fb61f2ed5d4db9a2a2979d6a9faa64 not found: ID does not exist" Nov 04 01:16:58.927222 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:58.927182 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-648b7c5784-dn7sb"] Nov 04 01:16:58.930361 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:58.930337 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-648b7c5784-dn7sb"] Nov 04 01:16:59.099824 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:59.099798 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-vvm6j"] Nov 04 01:16:59.101176 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:16:59.101152 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e927034_cbc7_47c7_919f_a0254565ac99.slice/crio-0e41ce5d9c1df82fe66b4b62700ca4f037f24e3058be1e0c38260ee7cbd3b196 WatchSource:0}: Error finding container 0e41ce5d9c1df82fe66b4b62700ca4f037f24e3058be1e0c38260ee7cbd3b196: Status 404 returned error can't find the container with id 0e41ce5d9c1df82fe66b4b62700ca4f037f24e3058be1e0c38260ee7cbd3b196 Nov 04 01:16:59.175953 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:59.175908 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd2407fe-d5a7-4207-8e0a-1bc1980abeb0" path="/var/lib/kubelet/pods/bd2407fe-d5a7-4207-8e0a-1bc1980abeb0/volumes" Nov 04 01:16:59.900909 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:16:59.900877 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-vvm6j" event={"ID":"5e927034-cbc7-47c7-919f-a0254565ac99","Type":"ContainerStarted","Data":"0e41ce5d9c1df82fe66b4b62700ca4f037f24e3058be1e0c38260ee7cbd3b196"} Nov 04 01:17:03.628937 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:03.628886 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-ch84f"] Nov 04 01:17:03.632699 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:03.632681 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-ch84f" Nov 04 01:17:03.642698 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:03.642619 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-ch84f"] Nov 04 01:17:03.659248 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:03.659202 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-qrllg"] Nov 04 01:17:03.663612 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:03.663590 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qrllg" Nov 04 01:17:03.672085 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:03.672061 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-dtthf\"" Nov 04 01:17:03.673747 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:03.673723 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-qrllg"] Nov 04 01:17:03.722407 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:03.722381 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 04 01:17:03.753986 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:03.753949 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a4edef8-b6bd-4c7e-a371-678894caee48-plugins-config-volume\") pod \"payload-processing-7bb9489676-ch84f\" (UID: \"9a4edef8-b6bd-4c7e-a371-678894caee48\") " pod="openshift-ingress/payload-processing-7bb9489676-ch84f" Nov 04 01:17:03.753986 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:03.753982 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4jlh\" (UniqueName: \"kubernetes.io/projected/07b7d302-ff11-4c35-a56a-040c692c85f9-kube-api-access-t4jlh\") pod \"payload-pre-processing-7b98bb678c-qrllg\" (UID: \"07b7d302-ff11-4c35-a56a-040c692c85f9\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qrllg" Nov 04 01:17:03.754182 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:03.754037 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc8rq\" (UniqueName: \"kubernetes.io/projected/9a4edef8-b6bd-4c7e-a371-678894caee48-kube-api-access-tc8rq\") pod \"payload-processing-7bb9489676-ch84f\" (UID: \"9a4edef8-b6bd-4c7e-a371-678894caee48\") " pod="openshift-ingress/payload-processing-7bb9489676-ch84f" Nov 04 01:17:03.754182 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:03.754124 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b7d302-ff11-4c35-a56a-040c692c85f9-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-qrllg\" (UID: \"07b7d302-ff11-4c35-a56a-040c692c85f9\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qrllg" Nov 04 01:17:03.855950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:03.855314 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a4edef8-b6bd-4c7e-a371-678894caee48-plugins-config-volume\") pod \"payload-processing-7bb9489676-ch84f\" (UID: \"9a4edef8-b6bd-4c7e-a371-678894caee48\") " pod="openshift-ingress/payload-processing-7bb9489676-ch84f" Nov 04 01:17:03.855950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:03.855361 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4jlh\" (UniqueName: \"kubernetes.io/projected/07b7d302-ff11-4c35-a56a-040c692c85f9-kube-api-access-t4jlh\") pod \"payload-pre-processing-7b98bb678c-qrllg\" (UID: \"07b7d302-ff11-4c35-a56a-040c692c85f9\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qrllg" Nov 04 01:17:03.855950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:03.855435 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc8rq\" (UniqueName: \"kubernetes.io/projected/9a4edef8-b6bd-4c7e-a371-678894caee48-kube-api-access-tc8rq\") pod \"payload-processing-7bb9489676-ch84f\" (UID: \"9a4edef8-b6bd-4c7e-a371-678894caee48\") " pod="openshift-ingress/payload-processing-7bb9489676-ch84f" Nov 04 01:17:03.855950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:03.855467 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b7d302-ff11-4c35-a56a-040c692c85f9-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-qrllg\" (UID: \"07b7d302-ff11-4c35-a56a-040c692c85f9\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qrllg" Nov 04 01:17:03.862957 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:03.856480 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b7d302-ff11-4c35-a56a-040c692c85f9-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-qrllg\" (UID: \"07b7d302-ff11-4c35-a56a-040c692c85f9\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qrllg" Nov 04 01:17:03.862957 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:03.857184 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a4edef8-b6bd-4c7e-a371-678894caee48-plugins-config-volume\") pod \"payload-processing-7bb9489676-ch84f\" (UID: \"9a4edef8-b6bd-4c7e-a371-678894caee48\") " pod="openshift-ingress/payload-processing-7bb9489676-ch84f" Nov 04 01:17:03.867674 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:03.867644 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4jlh\" (UniqueName: \"kubernetes.io/projected/07b7d302-ff11-4c35-a56a-040c692c85f9-kube-api-access-t4jlh\") pod \"payload-pre-processing-7b98bb678c-qrllg\" (UID: \"07b7d302-ff11-4c35-a56a-040c692c85f9\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qrllg" Nov 04 01:17:03.870956 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:03.870895 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc8rq\" (UniqueName: \"kubernetes.io/projected/9a4edef8-b6bd-4c7e-a371-678894caee48-kube-api-access-tc8rq\") pod \"payload-processing-7bb9489676-ch84f\" (UID: \"9a4edef8-b6bd-4c7e-a371-678894caee48\") " pod="openshift-ingress/payload-processing-7bb9489676-ch84f" Nov 04 01:17:03.925171 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:03.925144 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-vvm6j" event={"ID":"5e927034-cbc7-47c7-919f-a0254565ac99","Type":"ContainerStarted","Data":"c54fd55ece06bb5d760c548f1588c5c0f6ad4efecbcae8d84edf1c36aa0f0cd8"} Nov 04 01:17:03.925332 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:03.925233 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-vvm6j" Nov 04 01:17:03.943579 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:03.943521 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-vvm6j" podStartSLOduration=2.185083757 podStartE2EDuration="5.943505393s" totalImagesPullingTime="3.758421636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:03.941569378 +0000 UTC m=+1108.883418280" watchObservedRunningTime="2026-11-04 01:17:03.943505393 +0000 UTC m=+1108.885354296" Nov 04 01:17:03.947862 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:03.947832 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-ch84f" Nov 04 01:17:03.981588 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:03.981561 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qrllg" Nov 04 01:17:04.226564 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:04.226536 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-ch84f"] Nov 04 01:17:04.227602 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:17:04.227578 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a4edef8_b6bd_4c7e_a371_678894caee48.slice/crio-dc45dc92c7ae927739d402ed944f0f0857e8a8dc0c91697df0d128c764feaace WatchSource:0}: Error finding container dc45dc92c7ae927739d402ed944f0f0857e8a8dc0c91697df0d128c764feaace: Status 404 returned error can't find the container with id dc45dc92c7ae927739d402ed944f0f0857e8a8dc0c91697df0d128c764feaace Nov 04 01:17:04.267014 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:04.266910 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-qrllg"] Nov 04 01:17:04.281602 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:17:04.277635 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b7d302_ff11_4c35_a56a_040c692c85f9.slice/crio-e2c17465f5f9621883b6641932784140066ccf04b665acd6b2bdf84e17cc2855 WatchSource:0}: Error finding container e2c17465f5f9621883b6641932784140066ccf04b665acd6b2bdf84e17cc2855: Status 404 returned error can't find the container with id e2c17465f5f9621883b6641932784140066ccf04b665acd6b2bdf84e17cc2855 Nov 04 01:17:04.931825 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:04.931792 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qrllg" event={"ID":"07b7d302-ff11-4c35-a56a-040c692c85f9","Type":"ContainerStarted","Data":"e2c17465f5f9621883b6641932784140066ccf04b665acd6b2bdf84e17cc2855"} Nov 04 01:17:04.933226 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:04.933199 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-ch84f" event={"ID":"9a4edef8-b6bd-4c7e-a371-678894caee48","Type":"ContainerStarted","Data":"dc45dc92c7ae927739d402ed944f0f0857e8a8dc0c91697df0d128c764feaace"} Nov 04 01:17:06.490467 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:17:06.490391 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:17:06.949355 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:06.949322 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qrllg" event={"ID":"07b7d302-ff11-4c35-a56a-040c692c85f9","Type":"ContainerStarted","Data":"cd4570f728060c2df356de65d47c699403c70037ca6d06f1b452c6a024989fba"} Nov 04 01:17:06.949480 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:06.949441 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qrllg" Nov 04 01:17:06.951019 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:06.950994 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-ch84f" event={"ID":"9a4edef8-b6bd-4c7e-a371-678894caee48","Type":"ContainerStarted","Data":"c3724cab106ae1f85694a80676f80187611eefb1a737427149cd7a1c9cc322f9"} Nov 04 01:17:06.951142 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:06.951128 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7bb9489676-ch84f" Nov 04 01:17:06.967956 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:06.967903 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qrllg" podStartSLOduration=1.444944026 podStartE2EDuration="3.967890144s" totalImagesPullingTime="2.522946118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:06.965552356 +0000 UTC m=+1111.907401261" watchObservedRunningTime="2026-11-04 01:17:06.967890144 +0000 UTC m=+1111.909739099" Nov 04 01:17:06.985613 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:06.985576 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7bb9489676-ch84f" podStartSLOduration=1.44399754 podStartE2EDuration="3.985563864s" totalImagesPullingTime="2.541566324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:06.983448976 +0000 UTC m=+1111.925297883" watchObservedRunningTime="2026-11-04 01:17:06.985563864 +0000 UTC m=+1111.927412766" Nov 04 01:17:14.936139 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:14.936109 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-vvm6j" Nov 04 01:17:15.191651 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:15.191576 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6556489bf6-qrnzz"] Nov 04 01:17:15.195455 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:15.195435 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6556489bf6-qrnzz" Nov 04 01:17:15.203386 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:15.203305 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6556489bf6-qrnzz"] Nov 04 01:17:15.267292 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:15.267257 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnn7d\" (UniqueName: \"kubernetes.io/projected/b967836c-ea0e-4864-aee6-d76adbaa0656-kube-api-access-hnn7d\") pod \"ai-gateway-controller-6556489bf6-qrnzz\" (UID: \"b967836c-ea0e-4864-aee6-d76adbaa0656\") " pod="opendatahub/ai-gateway-controller-6556489bf6-qrnzz" Nov 04 01:17:15.267775 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:15.267755 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b967836c-ea0e-4864-aee6-d76adbaa0656-metrics-certs\") pod \"ai-gateway-controller-6556489bf6-qrnzz\" (UID: \"b967836c-ea0e-4864-aee6-d76adbaa0656\") " pod="opendatahub/ai-gateway-controller-6556489bf6-qrnzz" Nov 04 01:17:15.368862 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:15.368836 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnn7d\" (UniqueName: \"kubernetes.io/projected/b967836c-ea0e-4864-aee6-d76adbaa0656-kube-api-access-hnn7d\") pod \"ai-gateway-controller-6556489bf6-qrnzz\" (UID: \"b967836c-ea0e-4864-aee6-d76adbaa0656\") " pod="opendatahub/ai-gateway-controller-6556489bf6-qrnzz" Nov 04 01:17:15.368987 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:15.368953 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b967836c-ea0e-4864-aee6-d76adbaa0656-metrics-certs\") pod \"ai-gateway-controller-6556489bf6-qrnzz\" (UID: \"b967836c-ea0e-4864-aee6-d76adbaa0656\") " pod="opendatahub/ai-gateway-controller-6556489bf6-qrnzz" Nov 04 01:17:15.371451 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:15.371429 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b967836c-ea0e-4864-aee6-d76adbaa0656-metrics-certs\") pod \"ai-gateway-controller-6556489bf6-qrnzz\" (UID: \"b967836c-ea0e-4864-aee6-d76adbaa0656\") " pod="opendatahub/ai-gateway-controller-6556489bf6-qrnzz" Nov 04 01:17:15.379895 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:15.379619 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnn7d\" (UniqueName: \"kubernetes.io/projected/b967836c-ea0e-4864-aee6-d76adbaa0656-kube-api-access-hnn7d\") pod \"ai-gateway-controller-6556489bf6-qrnzz\" (UID: \"b967836c-ea0e-4864-aee6-d76adbaa0656\") " pod="opendatahub/ai-gateway-controller-6556489bf6-qrnzz" Nov 04 01:17:15.514971 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:15.514886 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6556489bf6-qrnzz" Nov 04 01:17:15.787686 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:15.787664 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6556489bf6-qrnzz"] Nov 04 01:17:15.793615 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:17:15.790070 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb967836c_ea0e_4864_aee6_d76adbaa0656.slice/crio-0e2d8af9ed684c8323af6b9efec9661330148fd301f2511089e47750db9bb7cd WatchSource:0}: Error finding container 0e2d8af9ed684c8323af6b9efec9661330148fd301f2511089e47750db9bb7cd: Status 404 returned error can't find the container with id 0e2d8af9ed684c8323af6b9efec9661330148fd301f2511089e47750db9bb7cd Nov 04 01:17:15.996993 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:15.996958 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6556489bf6-qrnzz" event={"ID":"b967836c-ea0e-4864-aee6-d76adbaa0656","Type":"ContainerStarted","Data":"0e2d8af9ed684c8323af6b9efec9661330148fd301f2511089e47750db9bb7cd"} Nov 04 01:17:17.004539 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:17.004507 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6556489bf6-qrnzz" event={"ID":"b967836c-ea0e-4864-aee6-d76adbaa0656","Type":"ContainerStarted","Data":"288347eddb40f3e336a8b60bf079b1a767efbf192a75674b9d1db0d24072209c"} Nov 04 01:17:17.004946 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:17.004638 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6556489bf6-qrnzz" Nov 04 01:17:17.034399 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:17.034354 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6556489bf6-qrnzz" podStartSLOduration=1.547518564 podStartE2EDuration="2.034341034s" totalImagesPullingTime="486.82247ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:17.03389288 +0000 UTC m=+1121.975741784" watchObservedRunningTime="2026-11-04 01:17:17.034341034 +0000 UTC m=+1121.976189936" Nov 04 01:17:17.959804 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:17.959776 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qrllg" Nov 04 01:17:17.959985 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:17.959839 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7bb9489676-ch84f" Nov 04 01:17:28.014519 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:28.014488 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6556489bf6-qrnzz" Nov 04 01:17:28.058958 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:28.058911 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-vvm6j"] Nov 04 01:17:28.059157 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:28.059117 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-vvm6j" podUID="5e927034-cbc7-47c7-919f-a0254565ac99" containerName="manager" containerID="cri-o://c54fd55ece06bb5d760c548f1588c5c0f6ad4efecbcae8d84edf1c36aa0f0cd8" gracePeriod=30 Nov 04 01:17:28.313695 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:28.313672 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-vvm6j" Nov 04 01:17:28.390833 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:28.390801 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e927034-cbc7-47c7-919f-a0254565ac99-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e927034-cbc7-47c7-919f-a0254565ac99-metrics-certs\") pod \"5e927034-cbc7-47c7-919f-a0254565ac99\" (UID: \"5e927034-cbc7-47c7-919f-a0254565ac99\") " Nov 04 01:17:28.391040 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:28.390876 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e927034-cbc7-47c7-919f-a0254565ac99-kube-api-access-rkhgn\" (UniqueName: \"kubernetes.io/projected/5e927034-cbc7-47c7-919f-a0254565ac99-kube-api-access-rkhgn\") pod \"5e927034-cbc7-47c7-919f-a0254565ac99\" (UID: \"5e927034-cbc7-47c7-919f-a0254565ac99\") " Nov 04 01:17:28.393107 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:28.393076 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e927034-cbc7-47c7-919f-a0254565ac99-metrics-certs" pod "5e927034-cbc7-47c7-919f-a0254565ac99" (UID: "5e927034-cbc7-47c7-919f-a0254565ac99"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:17:28.393405 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:28.393380 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e927034-cbc7-47c7-919f-a0254565ac99-kube-api-access-rkhgn" pod "5e927034-cbc7-47c7-919f-a0254565ac99" (UID: "5e927034-cbc7-47c7-919f-a0254565ac99"). InnerVolumeSpecName "kube-api-access-rkhgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:17:28.491881 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:28.491857 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkhgn\" (UniqueName: \"kubernetes.io/projected/5e927034-cbc7-47c7-919f-a0254565ac99-kube-api-access-rkhgn\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:17:28.491881 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:28.491881 7059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e927034-cbc7-47c7-919f-a0254565ac99-metrics-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:17:29.082075 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:29.082039 7059 generic.go:356] "Generic (PLEG): container finished" podID="5e927034-cbc7-47c7-919f-a0254565ac99" containerID="c54fd55ece06bb5d760c548f1588c5c0f6ad4efecbcae8d84edf1c36aa0f0cd8" exitCode=0 Nov 04 01:17:29.082457 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:29.082085 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-vvm6j" Nov 04 01:17:29.082457 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:29.082094 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-vvm6j" event={"ID":"5e927034-cbc7-47c7-919f-a0254565ac99","Type":"ContainerDied","Data":"c54fd55ece06bb5d760c548f1588c5c0f6ad4efecbcae8d84edf1c36aa0f0cd8"} Nov 04 01:17:29.082457 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:29.082146 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-vvm6j" event={"ID":"5e927034-cbc7-47c7-919f-a0254565ac99","Type":"ContainerDied","Data":"0e41ce5d9c1df82fe66b4b62700ca4f037f24e3058be1e0c38260ee7cbd3b196"} Nov 04 01:17:29.082457 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:29.082175 7059 scope.go:122] "RemoveContainer" containerID="c54fd55ece06bb5d760c548f1588c5c0f6ad4efecbcae8d84edf1c36aa0f0cd8" Nov 04 01:17:29.099019 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:29.098999 7059 scope.go:122] "RemoveContainer" containerID="c54fd55ece06bb5d760c548f1588c5c0f6ad4efecbcae8d84edf1c36aa0f0cd8" Nov 04 01:17:29.099280 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:17:29.099258 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c54fd55ece06bb5d760c548f1588c5c0f6ad4efecbcae8d84edf1c36aa0f0cd8\": container with ID starting with c54fd55ece06bb5d760c548f1588c5c0f6ad4efecbcae8d84edf1c36aa0f0cd8 not found: ID does not exist" containerID="c54fd55ece06bb5d760c548f1588c5c0f6ad4efecbcae8d84edf1c36aa0f0cd8" Nov 04 01:17:29.099349 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:29.099288 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c54fd55ece06bb5d760c548f1588c5c0f6ad4efecbcae8d84edf1c36aa0f0cd8"} err="failed to get container status \"c54fd55ece06bb5d760c548f1588c5c0f6ad4efecbcae8d84edf1c36aa0f0cd8\": rpc error: code = NotFound desc = could not find container \"c54fd55ece06bb5d760c548f1588c5c0f6ad4efecbcae8d84edf1c36aa0f0cd8\": container with ID starting with c54fd55ece06bb5d760c548f1588c5c0f6ad4efecbcae8d84edf1c36aa0f0cd8 not found: ID does not exist" Nov 04 01:17:29.108716 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:29.108688 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-vvm6j"] Nov 04 01:17:29.110570 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:29.110551 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-vvm6j"] Nov 04 01:17:29.175894 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:17:29.175869 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e927034-cbc7-47c7-919f-a0254565ac99" path="/var/lib/kubelet/pods/5e927034-cbc7-47c7-919f-a0254565ac99/volumes" Nov 04 01:17:36.479125 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:17:36.479090 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:18:02.229115 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.229057 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-9c8bf5-9758b5cf-9hsn2"] Nov 04 01:18:02.229705 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.229612 7059 cpu_manager.go:414] "containerMap: removing container" podUID="5e927034-cbc7-47c7-919f-a0254565ac99" containerName="manager" Nov 04 01:18:02.229705 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.229632 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e927034-cbc7-47c7-919f-a0254565ac99" containerName="manager" Nov 04 01:18:02.229819 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.229756 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e927034-cbc7-47c7-919f-a0254565ac99" containerName="manager" Nov 04 01:18:02.233106 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.233085 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9c8bf5-9758b5cf-9hsn2" Nov 04 01:18:02.246051 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.245782 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-9c8bf5\"" Nov 04 01:18:02.246051 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.245985 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-9c8bf5-dockercfg-nh829\"" Nov 04 01:18:02.247472 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.247447 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-9c8bf5-9758b5cf-9hsn2"] Nov 04 01:18:02.273020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.272991 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-9c8bf5-646c6d9774-4862q"] Nov 04 01:18:02.276671 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.276654 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-9c8bf5-646c6d9774-4862q" Nov 04 01:18:02.289431 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.289406 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-9c8bf5-646c6d9774-4862q"] Nov 04 01:18:02.304417 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.304388 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k4c2\" (UniqueName: \"kubernetes.io/projected/142d5751-9524-4f5c-8072-d05d5ca426ad-kube-api-access-6k4c2\") pod \"payload-pre-processing-e2e-worker-w5-9c8bf5-9758b5cf-9hsn2\" (UID: \"142d5751-9524-4f5c-8072-d05d5ca426ad\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9c8bf5-9758b5cf-9hsn2" Nov 04 01:18:02.304556 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.304422 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/142d5751-9524-4f5c-8072-d05d5ca426ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-9c8bf5-9758b5cf-9hsn2\" (UID: \"142d5751-9524-4f5c-8072-d05d5ca426ad\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9c8bf5-9758b5cf-9hsn2" Nov 04 01:18:02.304556 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.304450 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r25h8\" (UniqueName: \"kubernetes.io/projected/158365b9-7071-4a6d-b2a4-f6b56d795ec1-kube-api-access-r25h8\") pod \"payload-processing-e2e-worker-w5-9c8bf5-646c6d9774-4862q\" (UID: \"158365b9-7071-4a6d-b2a4-f6b56d795ec1\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-9c8bf5-646c6d9774-4862q" Nov 04 01:18:02.304689 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.304641 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/158365b9-7071-4a6d-b2a4-f6b56d795ec1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-9c8bf5-646c6d9774-4862q\" (UID: \"158365b9-7071-4a6d-b2a4-f6b56d795ec1\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-9c8bf5-646c6d9774-4862q" Nov 04 01:18:02.405522 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.405491 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/158365b9-7071-4a6d-b2a4-f6b56d795ec1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-9c8bf5-646c6d9774-4862q\" (UID: \"158365b9-7071-4a6d-b2a4-f6b56d795ec1\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-9c8bf5-646c6d9774-4862q" Nov 04 01:18:02.405722 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.405566 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k4c2\" (UniqueName: \"kubernetes.io/projected/142d5751-9524-4f5c-8072-d05d5ca426ad-kube-api-access-6k4c2\") pod \"payload-pre-processing-e2e-worker-w5-9c8bf5-9758b5cf-9hsn2\" (UID: \"142d5751-9524-4f5c-8072-d05d5ca426ad\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9c8bf5-9758b5cf-9hsn2" Nov 04 01:18:02.405722 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.405592 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/142d5751-9524-4f5c-8072-d05d5ca426ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-9c8bf5-9758b5cf-9hsn2\" (UID: \"142d5751-9524-4f5c-8072-d05d5ca426ad\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9c8bf5-9758b5cf-9hsn2" Nov 04 01:18:02.405858 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.405753 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r25h8\" (UniqueName: \"kubernetes.io/projected/158365b9-7071-4a6d-b2a4-f6b56d795ec1-kube-api-access-r25h8\") pod \"payload-processing-e2e-worker-w5-9c8bf5-646c6d9774-4862q\" (UID: \"158365b9-7071-4a6d-b2a4-f6b56d795ec1\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-9c8bf5-646c6d9774-4862q" Nov 04 01:18:02.406351 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.406328 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/158365b9-7071-4a6d-b2a4-f6b56d795ec1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-9c8bf5-646c6d9774-4862q\" (UID: \"158365b9-7071-4a6d-b2a4-f6b56d795ec1\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-9c8bf5-646c6d9774-4862q" Nov 04 01:18:02.406455 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.406376 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/142d5751-9524-4f5c-8072-d05d5ca426ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-9c8bf5-9758b5cf-9hsn2\" (UID: \"142d5751-9524-4f5c-8072-d05d5ca426ad\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9c8bf5-9758b5cf-9hsn2" Nov 04 01:18:02.414476 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.414451 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r25h8\" (UniqueName: \"kubernetes.io/projected/158365b9-7071-4a6d-b2a4-f6b56d795ec1-kube-api-access-r25h8\") pod \"payload-processing-e2e-worker-w5-9c8bf5-646c6d9774-4862q\" (UID: \"158365b9-7071-4a6d-b2a4-f6b56d795ec1\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-9c8bf5-646c6d9774-4862q" Nov 04 01:18:02.414595 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.414574 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k4c2\" (UniqueName: \"kubernetes.io/projected/142d5751-9524-4f5c-8072-d05d5ca426ad-kube-api-access-6k4c2\") pod \"payload-pre-processing-e2e-worker-w5-9c8bf5-9758b5cf-9hsn2\" (UID: \"142d5751-9524-4f5c-8072-d05d5ca426ad\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9c8bf5-9758b5cf-9hsn2" Nov 04 01:18:02.548634 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.548554 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9c8bf5-9758b5cf-9hsn2" Nov 04 01:18:02.591554 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.591319 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-9c8bf5-646c6d9774-4862q" Nov 04 01:18:02.605422 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.604757 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb"] Nov 04 01:18:02.619706 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.609866 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb" Nov 04 01:18:02.619706 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.619592 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb"] Nov 04 01:18:02.620037 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.619863 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nb79d\"" Nov 04 01:18:02.626062 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.620514 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 04 01:18:02.626062 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.620595 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-099dd9-serving-cert\"" Nov 04 01:18:02.723905 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.723875 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj8lw\" (UniqueName: \"kubernetes.io/projected/ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7-kube-api-access-vj8lw\") pod \"maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb\" (UID: \"ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb" Nov 04 01:18:02.724094 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.723916 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7-maas-api-tls\") pod \"maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb\" (UID: \"ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb" Nov 04 01:18:02.724162 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.724108 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7-metrics-certs\") pod \"maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb\" (UID: \"ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb" Nov 04 01:18:02.724245 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.724227 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb\" (UID: \"ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb" Nov 04 01:18:02.810455 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.810376 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-099dd9-775897687b-7z7mn"] Nov 04 01:18:02.814268 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.814241 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-099dd9-775897687b-7z7mn" Nov 04 01:18:02.821056 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.821027 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-099dd9-775897687b-7z7mn"] Nov 04 01:18:02.821488 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.821133 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-099dd9\"" Nov 04 01:18:02.821488 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.821235 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-099dd9-dockercfg-6spm4\"" Nov 04 01:18:02.825228 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.825200 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj8lw\" (UniqueName: \"kubernetes.io/projected/ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7-kube-api-access-vj8lw\") pod \"maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb\" (UID: \"ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb" Nov 04 01:18:02.825404 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.825256 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7-maas-api-tls\") pod \"maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb\" (UID: \"ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb" Nov 04 01:18:02.825527 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.825390 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7-metrics-certs\") pod \"maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb\" (UID: \"ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb" Nov 04 01:18:02.825628 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.825545 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb\" (UID: \"ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb" Nov 04 01:18:02.826727 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.826594 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb\" (UID: \"ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb" Nov 04 01:18:02.831535 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.831509 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7-metrics-certs\") pod \"maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb\" (UID: \"ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb" Nov 04 01:18:02.832997 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.832967 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7-maas-api-tls\") pod \"maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb\" (UID: \"ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb" Nov 04 01:18:02.839009 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.838986 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-099dd9-5f4444cd6c-g7kl6"] Nov 04 01:18:02.843268 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.843249 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-099dd9-5f4444cd6c-g7kl6" Nov 04 01:18:02.846048 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.846023 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj8lw\" (UniqueName: \"kubernetes.io/projected/ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7-kube-api-access-vj8lw\") pod \"maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb\" (UID: \"ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb" Nov 04 01:18:02.860981 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.853483 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-099dd9-5f4444cd6c-g7kl6"] Nov 04 01:18:02.861848 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.861823 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-9c8bf5-9758b5cf-9hsn2"] Nov 04 01:18:02.908593 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.908571 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-9c8bf5-646c6d9774-4862q"] Nov 04 01:18:02.911423 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:18:02.911391 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod158365b9_7071_4a6d_b2a4_f6b56d795ec1.slice/crio-35626fdc097ee9ee026a4717d7ad078fef2dda92283c808b77d371bddac1c330 WatchSource:0}: Error finding container 35626fdc097ee9ee026a4717d7ad078fef2dda92283c808b77d371bddac1c330: Status 404 returned error can't find the container with id 35626fdc097ee9ee026a4717d7ad078fef2dda92283c808b77d371bddac1c330 Nov 04 01:18:02.927315 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.927280 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5msw\" (UniqueName: \"kubernetes.io/projected/2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d-kube-api-access-v5msw\") pod \"payload-pre-processing-e2e-worker-w4-099dd9-5f4444cd6c-g7kl6\" (UID: \"2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-099dd9-5f4444cd6c-g7kl6" Nov 04 01:18:02.927438 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.927383 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53fd8082-374a-44b8-bf5d-25b56fdc7c10-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-099dd9-775897687b-7z7mn\" (UID: \"53fd8082-374a-44b8-bf5d-25b56fdc7c10\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-099dd9-775897687b-7z7mn" Nov 04 01:18:02.927438 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.927417 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fvmg\" (UniqueName: \"kubernetes.io/projected/53fd8082-374a-44b8-bf5d-25b56fdc7c10-kube-api-access-5fvmg\") pod \"payload-processing-e2e-worker-w4-099dd9-775897687b-7z7mn\" (UID: \"53fd8082-374a-44b8-bf5d-25b56fdc7c10\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-099dd9-775897687b-7z7mn" Nov 04 01:18:02.927555 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.927446 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-099dd9-5f4444cd6c-g7kl6\" (UID: \"2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-099dd9-5f4444cd6c-g7kl6" Nov 04 01:18:02.956564 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:02.956262 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb" Nov 04 01:18:03.029844 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.029459 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5msw\" (UniqueName: \"kubernetes.io/projected/2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d-kube-api-access-v5msw\") pod \"payload-pre-processing-e2e-worker-w4-099dd9-5f4444cd6c-g7kl6\" (UID: \"2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-099dd9-5f4444cd6c-g7kl6" Nov 04 01:18:03.029844 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.029499 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53fd8082-374a-44b8-bf5d-25b56fdc7c10-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-099dd9-775897687b-7z7mn\" (UID: \"53fd8082-374a-44b8-bf5d-25b56fdc7c10\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-099dd9-775897687b-7z7mn" Nov 04 01:18:03.029844 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.029519 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fvmg\" (UniqueName: \"kubernetes.io/projected/53fd8082-374a-44b8-bf5d-25b56fdc7c10-kube-api-access-5fvmg\") pod \"payload-processing-e2e-worker-w4-099dd9-775897687b-7z7mn\" (UID: \"53fd8082-374a-44b8-bf5d-25b56fdc7c10\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-099dd9-775897687b-7z7mn" Nov 04 01:18:03.029844 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.029539 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-099dd9-5f4444cd6c-g7kl6\" (UID: \"2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-099dd9-5f4444cd6c-g7kl6" Nov 04 01:18:03.030547 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.030495 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-099dd9-5f4444cd6c-g7kl6\" (UID: \"2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-099dd9-5f4444cd6c-g7kl6" Nov 04 01:18:03.030985 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.030948 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53fd8082-374a-44b8-bf5d-25b56fdc7c10-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-099dd9-775897687b-7z7mn\" (UID: \"53fd8082-374a-44b8-bf5d-25b56fdc7c10\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-099dd9-775897687b-7z7mn" Nov 04 01:18:03.048661 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.048639 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fvmg\" (UniqueName: \"kubernetes.io/projected/53fd8082-374a-44b8-bf5d-25b56fdc7c10-kube-api-access-5fvmg\") pod \"payload-processing-e2e-worker-w4-099dd9-775897687b-7z7mn\" (UID: \"53fd8082-374a-44b8-bf5d-25b56fdc7c10\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-099dd9-775897687b-7z7mn" Nov 04 01:18:03.052964 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.052885 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5msw\" (UniqueName: \"kubernetes.io/projected/2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d-kube-api-access-v5msw\") pod \"payload-pre-processing-e2e-worker-w4-099dd9-5f4444cd6c-g7kl6\" (UID: \"2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-099dd9-5f4444cd6c-g7kl6" Nov 04 01:18:03.137897 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.137831 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-099dd9-775897687b-7z7mn" Nov 04 01:18:03.165304 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.165271 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-099dd9-5f4444cd6c-g7kl6" Nov 04 01:18:03.252724 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.252693 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb"] Nov 04 01:18:03.253249 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:18:03.253218 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba4d5980_b4ce_4e3e_b9f3_65f4c871eca7.slice/crio-489de3e40f8fddcaaef49147c81a4e614d19d6cbb1e8c910f0870cdc2a930ebb WatchSource:0}: Error finding container 489de3e40f8fddcaaef49147c81a4e614d19d6cbb1e8c910f0870cdc2a930ebb: Status 404 returned error can't find the container with id 489de3e40f8fddcaaef49147c81a4e614d19d6cbb1e8c910f0870cdc2a930ebb Nov 04 01:18:03.314970 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.313431 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-9c8bf5-646c6d9774-4862q" event={"ID":"158365b9-7071-4a6d-b2a4-f6b56d795ec1","Type":"ContainerStarted","Data":"635f93a606a4c10f2fe4d9f15a86fa07dcd9fabfeb5b131212eac77cfc4095d9"} Nov 04 01:18:03.314970 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.313492 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-9c8bf5-646c6d9774-4862q" event={"ID":"158365b9-7071-4a6d-b2a4-f6b56d795ec1","Type":"ContainerStarted","Data":"35626fdc097ee9ee026a4717d7ad078fef2dda92283c808b77d371bddac1c330"} Nov 04 01:18:03.314970 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.313602 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-9c8bf5-646c6d9774-4862q" Nov 04 01:18:03.314970 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.314744 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb" event={"ID":"ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7","Type":"ContainerStarted","Data":"489de3e40f8fddcaaef49147c81a4e614d19d6cbb1e8c910f0870cdc2a930ebb"} Nov 04 01:18:03.316432 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.316401 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9c8bf5-9758b5cf-9hsn2" event={"ID":"142d5751-9524-4f5c-8072-d05d5ca426ad","Type":"ContainerStarted","Data":"cbc20e0c0e789a803cca3793bf9f27ec3d0cc9841d566664fcff950c302a7a52"} Nov 04 01:18:03.316566 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.316453 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9c8bf5-9758b5cf-9hsn2" event={"ID":"142d5751-9524-4f5c-8072-d05d5ca426ad","Type":"ContainerStarted","Data":"a7c54f68c6097a30b5ec38cfa46ec9b784feff54cd472689024e20db32679679"} Nov 04 01:18:03.316566 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.316483 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9c8bf5-9758b5cf-9hsn2" Nov 04 01:18:03.336419 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.336377 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-9c8bf5-646c6d9774-4862q" podStartSLOduration=1.33636405 podStartE2EDuration="1.33636405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:03.334224443 +0000 UTC m=+1168.276073347" watchObservedRunningTime="2026-11-04 01:18:03.33636405 +0000 UTC m=+1168.278212953" Nov 04 01:18:03.356156 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.356115 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9c8bf5-9758b5cf-9hsn2" podStartSLOduration=1.356103596 podStartE2EDuration="1.356103596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:03.355281551 +0000 UTC m=+1168.297130454" watchObservedRunningTime="2026-11-04 01:18:03.356103596 +0000 UTC m=+1168.297952499" Nov 04 01:18:03.419022 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.418998 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-099dd9-775897687b-7z7mn"] Nov 04 01:18:03.421008 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:18:03.420978 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53fd8082_374a_44b8_bf5d_25b56fdc7c10.slice/crio-4667766d006134eb52bd983b2701f5165b7ea229f44fa52fa365d505140df591 WatchSource:0}: Error finding container 4667766d006134eb52bd983b2701f5165b7ea229f44fa52fa365d505140df591: Status 404 returned error can't find the container with id 4667766d006134eb52bd983b2701f5165b7ea229f44fa52fa365d505140df591 Nov 04 01:18:03.466725 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.466690 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-099dd9-5f4444cd6c-g7kl6"] Nov 04 01:18:03.480157 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:18:03.480123 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c5b91a9_d24f_4fa6_8df1_cb27aec9d47d.slice/crio-9e961f1952e7e64a6fb34b71ae0c7cf04502779a4dbfdcdbb56dbd962927665e WatchSource:0}: Error finding container 9e961f1952e7e64a6fb34b71ae0c7cf04502779a4dbfdcdbb56dbd962927665e: Status 404 returned error can't find the container with id 9e961f1952e7e64a6fb34b71ae0c7cf04502779a4dbfdcdbb56dbd962927665e Nov 04 01:18:03.558209 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.558182 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ca55c1-66d7c55b44-mm9qn"] Nov 04 01:18:03.564242 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.564223 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-ca55c1-66d7c55b44-mm9qn" Nov 04 01:18:03.568973 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.568937 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-ca55c1\"" Nov 04 01:18:03.569565 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.569543 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-ca55c1-dockercfg-svzxb\"" Nov 04 01:18:03.572325 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.572301 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ca55c1-66d7c55b44-mm9qn"] Nov 04 01:18:03.575448 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.575427 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ca55c1-5bb65b58bc-j9bkv"] Nov 04 01:18:03.579685 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.579651 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ca55c1-5bb65b58bc-j9bkv" Nov 04 01:18:03.594310 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.594276 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ca55c1-5bb65b58bc-j9bkv"] Nov 04 01:18:03.640016 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.639987 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f2ff\" (UniqueName: \"kubernetes.io/projected/9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5-kube-api-access-7f2ff\") pod \"payload-processing-e2e-worker-w3-ca55c1-66d7c55b44-mm9qn\" (UID: \"9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ca55c1-66d7c55b44-mm9qn" Nov 04 01:18:03.640159 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.640031 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h78pt\" (UniqueName: \"kubernetes.io/projected/3744cca1-904b-43ac-870b-c07ced58941c-kube-api-access-h78pt\") pod \"payload-pre-processing-e2e-worker-w3-ca55c1-5bb65b58bc-j9bkv\" (UID: \"3744cca1-904b-43ac-870b-c07ced58941c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ca55c1-5bb65b58bc-j9bkv" Nov 04 01:18:03.640159 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.640066 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3744cca1-904b-43ac-870b-c07ced58941c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-ca55c1-5bb65b58bc-j9bkv\" (UID: \"3744cca1-904b-43ac-870b-c07ced58941c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ca55c1-5bb65b58bc-j9bkv" Nov 04 01:18:03.640243 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.640209 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-ca55c1-66d7c55b44-mm9qn\" (UID: \"9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ca55c1-66d7c55b44-mm9qn" Nov 04 01:18:03.741773 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.741730 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f2ff\" (UniqueName: \"kubernetes.io/projected/9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5-kube-api-access-7f2ff\") pod \"payload-processing-e2e-worker-w3-ca55c1-66d7c55b44-mm9qn\" (UID: \"9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ca55c1-66d7c55b44-mm9qn" Nov 04 01:18:03.741773 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.741776 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h78pt\" (UniqueName: \"kubernetes.io/projected/3744cca1-904b-43ac-870b-c07ced58941c-kube-api-access-h78pt\") pod \"payload-pre-processing-e2e-worker-w3-ca55c1-5bb65b58bc-j9bkv\" (UID: \"3744cca1-904b-43ac-870b-c07ced58941c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ca55c1-5bb65b58bc-j9bkv" Nov 04 01:18:03.742055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.741810 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3744cca1-904b-43ac-870b-c07ced58941c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-ca55c1-5bb65b58bc-j9bkv\" (UID: \"3744cca1-904b-43ac-870b-c07ced58941c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ca55c1-5bb65b58bc-j9bkv" Nov 04 01:18:03.742055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.741869 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-ca55c1-66d7c55b44-mm9qn\" (UID: \"9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ca55c1-66d7c55b44-mm9qn" Nov 04 01:18:03.742699 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.742676 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3744cca1-904b-43ac-870b-c07ced58941c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-ca55c1-5bb65b58bc-j9bkv\" (UID: \"3744cca1-904b-43ac-870b-c07ced58941c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ca55c1-5bb65b58bc-j9bkv" Nov 04 01:18:03.742775 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.742679 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-ca55c1-66d7c55b44-mm9qn\" (UID: \"9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ca55c1-66d7c55b44-mm9qn" Nov 04 01:18:03.750231 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.750204 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f2ff\" (UniqueName: \"kubernetes.io/projected/9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5-kube-api-access-7f2ff\") pod \"payload-processing-e2e-worker-w3-ca55c1-66d7c55b44-mm9qn\" (UID: \"9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ca55c1-66d7c55b44-mm9qn" Nov 04 01:18:03.750337 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.750234 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h78pt\" (UniqueName: \"kubernetes.io/projected/3744cca1-904b-43ac-870b-c07ced58941c-kube-api-access-h78pt\") pod \"payload-pre-processing-e2e-worker-w3-ca55c1-5bb65b58bc-j9bkv\" (UID: \"3744cca1-904b-43ac-870b-c07ced58941c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ca55c1-5bb65b58bc-j9bkv" Nov 04 01:18:03.889113 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.889080 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-ca55c1-66d7c55b44-mm9qn" Nov 04 01:18:03.899981 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.899913 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ca55c1-5bb65b58bc-j9bkv" Nov 04 01:18:03.929394 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.927905 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn"] Nov 04 01:18:03.944719 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.936216 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn" Nov 04 01:18:03.950191 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.950045 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn"] Nov 04 01:18:03.952257 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:03.951639 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-ca55c1-serving-cert\"" Nov 04 01:18:04.044805 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.044771 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqggg\" (UniqueName: \"kubernetes.io/projected/facc3749-8125-4daa-956e-e521856cc7da-kube-api-access-sqggg\") pod \"maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn\" (UID: \"facc3749-8125-4daa-956e-e521856cc7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn" Nov 04 01:18:04.044977 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.044894 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/facc3749-8125-4daa-956e-e521856cc7da-metrics-certs\") pod \"maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn\" (UID: \"facc3749-8125-4daa-956e-e521856cc7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn" Nov 04 01:18:04.045213 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.045174 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/facc3749-8125-4daa-956e-e521856cc7da-maas-api-tls\") pod \"maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn\" (UID: \"facc3749-8125-4daa-956e-e521856cc7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn" Nov 04 01:18:04.045290 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.045235 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/facc3749-8125-4daa-956e-e521856cc7da-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn\" (UID: \"facc3749-8125-4daa-956e-e521856cc7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn" Nov 04 01:18:04.046680 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.046646 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-499ddd-68c97cbdc5-57gdf"] Nov 04 01:18:04.052707 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.052683 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-499ddd-68c97cbdc5-57gdf" Nov 04 01:18:04.059289 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.059265 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-499ddd\"" Nov 04 01:18:04.059289 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.059272 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-499ddd-dockercfg-4vsjj\"" Nov 04 01:18:04.062445 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.062418 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-499ddd-68c97cbdc5-57gdf"] Nov 04 01:18:04.089097 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.089056 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-499ddd-677fd64986-9c76n"] Nov 04 01:18:04.094310 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.094288 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-499ddd-677fd64986-9c76n" Nov 04 01:18:04.105438 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.104894 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-499ddd-677fd64986-9c76n"] Nov 04 01:18:04.151585 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.151547 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22a00871-73e9-4c9a-aaa6-9046b1004b9e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-499ddd-677fd64986-9c76n\" (UID: \"22a00871-73e9-4c9a-aaa6-9046b1004b9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-499ddd-677fd64986-9c76n" Nov 04 01:18:04.151982 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.151605 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/facc3749-8125-4daa-956e-e521856cc7da-maas-api-tls\") pod \"maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn\" (UID: \"facc3749-8125-4daa-956e-e521856cc7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn" Nov 04 01:18:04.151982 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.151651 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/facc3749-8125-4daa-956e-e521856cc7da-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn\" (UID: \"facc3749-8125-4daa-956e-e521856cc7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn" Nov 04 01:18:04.151982 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.151707 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5prz\" (UniqueName: \"kubernetes.io/projected/1510cc32-e22b-46f0-acfb-3e2994767889-kube-api-access-m5prz\") pod \"payload-pre-processing-e2e-worker-w6-499ddd-68c97cbdc5-57gdf\" (UID: \"1510cc32-e22b-46f0-acfb-3e2994767889\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-499ddd-68c97cbdc5-57gdf" Nov 04 01:18:04.151982 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.151746 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqggg\" (UniqueName: \"kubernetes.io/projected/facc3749-8125-4daa-956e-e521856cc7da-kube-api-access-sqggg\") pod \"maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn\" (UID: \"facc3749-8125-4daa-956e-e521856cc7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn" Nov 04 01:18:04.151982 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.151843 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1510cc32-e22b-46f0-acfb-3e2994767889-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-499ddd-68c97cbdc5-57gdf\" (UID: \"1510cc32-e22b-46f0-acfb-3e2994767889\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-499ddd-68c97cbdc5-57gdf" Nov 04 01:18:04.151982 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.151878 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zjnp\" (UniqueName: \"kubernetes.io/projected/22a00871-73e9-4c9a-aaa6-9046b1004b9e-kube-api-access-8zjnp\") pod \"payload-processing-e2e-worker-w6-499ddd-677fd64986-9c76n\" (UID: \"22a00871-73e9-4c9a-aaa6-9046b1004b9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-499ddd-677fd64986-9c76n" Nov 04 01:18:04.151982 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.151911 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/facc3749-8125-4daa-956e-e521856cc7da-metrics-certs\") pod \"maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn\" (UID: \"facc3749-8125-4daa-956e-e521856cc7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn" Nov 04 01:18:04.153964 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.153918 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/facc3749-8125-4daa-956e-e521856cc7da-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn\" (UID: \"facc3749-8125-4daa-956e-e521856cc7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn" Nov 04 01:18:04.159377 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.159349 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/facc3749-8125-4daa-956e-e521856cc7da-metrics-certs\") pod \"maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn\" (UID: \"facc3749-8125-4daa-956e-e521856cc7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn" Nov 04 01:18:04.160260 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.160234 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/facc3749-8125-4daa-956e-e521856cc7da-maas-api-tls\") pod \"maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn\" (UID: \"facc3749-8125-4daa-956e-e521856cc7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn" Nov 04 01:18:04.163562 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.163516 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqggg\" (UniqueName: \"kubernetes.io/projected/facc3749-8125-4daa-956e-e521856cc7da-kube-api-access-sqggg\") pod \"maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn\" (UID: \"facc3749-8125-4daa-956e-e521856cc7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn" Nov 04 01:18:04.192827 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.192795 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ca55c1-66d7c55b44-mm9qn"] Nov 04 01:18:04.197144 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:18:04.197115 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9798d70c_a8f1_4aa5_ba8e_41f902f4f6e5.slice/crio-05c7d80305f02937b39d9e1104d7dcb3c684016e1dc7d91261bb3ed68dfe1fa7 WatchSource:0}: Error finding container 05c7d80305f02937b39d9e1104d7dcb3c684016e1dc7d91261bb3ed68dfe1fa7: Status 404 returned error can't find the container with id 05c7d80305f02937b39d9e1104d7dcb3c684016e1dc7d91261bb3ed68dfe1fa7 Nov 04 01:18:04.217977 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.217951 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ca55c1-5bb65b58bc-j9bkv"] Nov 04 01:18:04.220715 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:18:04.220617 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3744cca1_904b_43ac_870b_c07ced58941c.slice/crio-67a3136505ed4734cece77181a3e4078eca57c4fc4d5bd248a2bdbd6c800ce4c WatchSource:0}: Error finding container 67a3136505ed4734cece77181a3e4078eca57c4fc4d5bd248a2bdbd6c800ce4c: Status 404 returned error can't find the container with id 67a3136505ed4734cece77181a3e4078eca57c4fc4d5bd248a2bdbd6c800ce4c Nov 04 01:18:04.253403 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.253296 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22a00871-73e9-4c9a-aaa6-9046b1004b9e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-499ddd-677fd64986-9c76n\" (UID: \"22a00871-73e9-4c9a-aaa6-9046b1004b9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-499ddd-677fd64986-9c76n" Nov 04 01:18:04.253978 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.253951 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5prz\" (UniqueName: \"kubernetes.io/projected/1510cc32-e22b-46f0-acfb-3e2994767889-kube-api-access-m5prz\") pod \"payload-pre-processing-e2e-worker-w6-499ddd-68c97cbdc5-57gdf\" (UID: \"1510cc32-e22b-46f0-acfb-3e2994767889\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-499ddd-68c97cbdc5-57gdf" Nov 04 01:18:04.254124 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.253994 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22a00871-73e9-4c9a-aaa6-9046b1004b9e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-499ddd-677fd64986-9c76n\" (UID: \"22a00871-73e9-4c9a-aaa6-9046b1004b9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-499ddd-677fd64986-9c76n" Nov 04 01:18:04.254124 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.254096 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1510cc32-e22b-46f0-acfb-3e2994767889-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-499ddd-68c97cbdc5-57gdf\" (UID: \"1510cc32-e22b-46f0-acfb-3e2994767889\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-499ddd-68c97cbdc5-57gdf" Nov 04 01:18:04.254265 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.254123 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zjnp\" (UniqueName: \"kubernetes.io/projected/22a00871-73e9-4c9a-aaa6-9046b1004b9e-kube-api-access-8zjnp\") pod \"payload-processing-e2e-worker-w6-499ddd-677fd64986-9c76n\" (UID: \"22a00871-73e9-4c9a-aaa6-9046b1004b9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-499ddd-677fd64986-9c76n" Nov 04 01:18:04.255412 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.255383 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1510cc32-e22b-46f0-acfb-3e2994767889-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-499ddd-68c97cbdc5-57gdf\" (UID: \"1510cc32-e22b-46f0-acfb-3e2994767889\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-499ddd-68c97cbdc5-57gdf" Nov 04 01:18:04.263473 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.263452 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn" Nov 04 01:18:04.263860 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.263841 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zjnp\" (UniqueName: \"kubernetes.io/projected/22a00871-73e9-4c9a-aaa6-9046b1004b9e-kube-api-access-8zjnp\") pod \"payload-processing-e2e-worker-w6-499ddd-677fd64986-9c76n\" (UID: \"22a00871-73e9-4c9a-aaa6-9046b1004b9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-499ddd-677fd64986-9c76n" Nov 04 01:18:04.264001 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.263985 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5prz\" (UniqueName: \"kubernetes.io/projected/1510cc32-e22b-46f0-acfb-3e2994767889-kube-api-access-m5prz\") pod \"payload-pre-processing-e2e-worker-w6-499ddd-68c97cbdc5-57gdf\" (UID: \"1510cc32-e22b-46f0-acfb-3e2994767889\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-499ddd-68c97cbdc5-57gdf" Nov 04 01:18:04.324628 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.324583 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-ca55c1-66d7c55b44-mm9qn" event={"ID":"9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5","Type":"ContainerStarted","Data":"1be4734bd41614ec07cb044311ed9b6f947a62b4f4b7d3cb7a161d589d06f316"} Nov 04 01:18:04.324813 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.324646 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-ca55c1-66d7c55b44-mm9qn" event={"ID":"9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5","Type":"ContainerStarted","Data":"05c7d80305f02937b39d9e1104d7dcb3c684016e1dc7d91261bb3ed68dfe1fa7"} Nov 04 01:18:04.325071 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.325007 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-ca55c1-66d7c55b44-mm9qn" Nov 04 01:18:04.332974 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.331866 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-099dd9-775897687b-7z7mn" event={"ID":"53fd8082-374a-44b8-bf5d-25b56fdc7c10","Type":"ContainerStarted","Data":"8e86037cc877b72234842a5c0634d757ef51f40298d36a2fb6d8f2532ab2b111"} Nov 04 01:18:04.332974 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.331960 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-099dd9-775897687b-7z7mn" event={"ID":"53fd8082-374a-44b8-bf5d-25b56fdc7c10","Type":"ContainerStarted","Data":"4667766d006134eb52bd983b2701f5165b7ea229f44fa52fa365d505140df591"} Nov 04 01:18:04.332974 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.332399 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-099dd9-775897687b-7z7mn" Nov 04 01:18:04.334092 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.334064 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ca55c1-5bb65b58bc-j9bkv" event={"ID":"3744cca1-904b-43ac-870b-c07ced58941c","Type":"ContainerStarted","Data":"015382e8cb8510b13cb6f135ba82a1f7ba8d9db491645e3932f0963c7f8cee7d"} Nov 04 01:18:04.334200 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.334113 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ca55c1-5bb65b58bc-j9bkv" event={"ID":"3744cca1-904b-43ac-870b-c07ced58941c","Type":"ContainerStarted","Data":"67a3136505ed4734cece77181a3e4078eca57c4fc4d5bd248a2bdbd6c800ce4c"} Nov 04 01:18:04.334200 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.334142 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ca55c1-5bb65b58bc-j9bkv" Nov 04 01:18:04.335692 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.335670 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-099dd9-5f4444cd6c-g7kl6" event={"ID":"2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d","Type":"ContainerStarted","Data":"5f811d7b3b328df2ce9530b84b4460a47ad2b8e2c1c08d6e8fb173f07557bcb2"} Nov 04 01:18:04.335776 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.335712 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-099dd9-5f4444cd6c-g7kl6" event={"ID":"2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d","Type":"ContainerStarted","Data":"9e961f1952e7e64a6fb34b71ae0c7cf04502779a4dbfdcdbb56dbd962927665e"} Nov 04 01:18:04.336173 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.336154 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-099dd9-5f4444cd6c-g7kl6" Nov 04 01:18:04.342222 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.342199 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb" event={"ID":"ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7","Type":"ContainerStarted","Data":"ba2c0c5c88b4526fc1c25203effa81a548b653ccf8cf8b057e96aad5788cec3b"} Nov 04 01:18:04.342306 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.342242 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb" Nov 04 01:18:04.350563 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.350515 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-ca55c1-66d7c55b44-mm9qn" podStartSLOduration=1.350499981 podStartE2EDuration="1.350499981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:04.349292063 +0000 UTC m=+1169.291140992" watchObservedRunningTime="2026-11-04 01:18:04.350499981 +0000 UTC m=+1169.292348887" Nov 04 01:18:04.380688 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.380645 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-499ddd-68c97cbdc5-57gdf" Nov 04 01:18:04.401894 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.401834 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-099dd9-775897687b-7z7mn" podStartSLOduration=2.401817281 podStartE2EDuration="2.401817281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:04.400195692 +0000 UTC m=+1169.342044594" watchObservedRunningTime="2026-11-04 01:18:04.401817281 +0000 UTC m=+1169.343666182" Nov 04 01:18:04.402380 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.402349 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb" podStartSLOduration=1.6806571209999999 podStartE2EDuration="2.402337985s" totalImagesPullingTime="721.680864ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:04.375433343 +0000 UTC m=+1169.317282250" watchObservedRunningTime="2026-11-04 01:18:04.402337985 +0000 UTC m=+1169.344186888" Nov 04 01:18:04.419517 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.419495 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-499ddd-677fd64986-9c76n" Nov 04 01:18:04.423792 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.423070 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-099dd9-5f4444cd6c-g7kl6" podStartSLOduration=2.423057983 podStartE2EDuration="2.423057983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:04.421783926 +0000 UTC m=+1169.363632842" watchObservedRunningTime="2026-11-04 01:18:04.423057983 +0000 UTC m=+1169.364906968" Nov 04 01:18:04.445125 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.443633 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ca55c1-5bb65b58bc-j9bkv" podStartSLOduration=1.443613482 podStartE2EDuration="1.443613482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:04.440970324 +0000 UTC m=+1169.382819228" watchObservedRunningTime="2026-11-04 01:18:04.443613482 +0000 UTC m=+1169.385462387" Nov 04 01:18:04.574403 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.574371 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn"] Nov 04 01:18:04.577844 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:18:04.577817 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfacc3749_8125_4daa_956e_e521856cc7da.slice/crio-37ab04886888dab059a554a1163e581285571810525ebb3b10ac33ae6c016fae WatchSource:0}: Error finding container 37ab04886888dab059a554a1163e581285571810525ebb3b10ac33ae6c016fae: Status 404 returned error can't find the container with id 37ab04886888dab059a554a1163e581285571810525ebb3b10ac33ae6c016fae Nov 04 01:18:04.642386 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.642358 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d7e45d-78c9c88f96-sggpl"] Nov 04 01:18:04.655333 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.655301 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-d7e45d-78c9c88f96-sggpl" Nov 04 01:18:04.667627 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.667590 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d7e45d-78c9c88f96-sggpl"] Nov 04 01:18:04.668988 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.668970 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-d7e45d\"" Nov 04 01:18:04.669479 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.669446 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-d7e45d-dockercfg-s4j5r\"" Nov 04 01:18:04.676524 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.676499 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d7e45d-7ff9858d99-qv8gj"] Nov 04 01:18:04.681207 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.681187 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d7e45d-7ff9858d99-qv8gj" Nov 04 01:18:04.691375 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.691354 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-499ddd-68c97cbdc5-57gdf"] Nov 04 01:18:04.697119 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.697084 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d7e45d-7ff9858d99-qv8gj"] Nov 04 01:18:04.698635 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:18:04.698612 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1510cc32_e22b_46f0_acfb_3e2994767889.slice/crio-5fe928c771a4d888b4a000b342e004349e19cd720f5db91dd8cc9b4ad3b9417e WatchSource:0}: Error finding container 5fe928c771a4d888b4a000b342e004349e19cd720f5db91dd8cc9b4ad3b9417e: Status 404 returned error can't find the container with id 5fe928c771a4d888b4a000b342e004349e19cd720f5db91dd8cc9b4ad3b9417e Nov 04 01:18:04.734231 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.734199 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-499ddd-677fd64986-9c76n"] Nov 04 01:18:04.735556 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:18:04.735532 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a00871_73e9_4c9a_aaa6_9046b1004b9e.slice/crio-77a4bc64fbd8dd01eeff26cbea3a7d518513f9b48baa0eeeb4bc111f082df62e WatchSource:0}: Error finding container 77a4bc64fbd8dd01eeff26cbea3a7d518513f9b48baa0eeeb4bc111f082df62e: Status 404 returned error can't find the container with id 77a4bc64fbd8dd01eeff26cbea3a7d518513f9b48baa0eeeb4bc111f082df62e Nov 04 01:18:04.774949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.760055 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwnf9\" (UniqueName: \"kubernetes.io/projected/507f1fa6-dc41-4285-9e97-941eb8e9f525-kube-api-access-nwnf9\") pod \"payload-pre-processing-e2e-worker-w2-d7e45d-7ff9858d99-qv8gj\" (UID: \"507f1fa6-dc41-4285-9e97-941eb8e9f525\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d7e45d-7ff9858d99-qv8gj" Nov 04 01:18:04.774949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.760115 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d090174e-7ec3-4c2c-a296-f15f9543ea36-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-d7e45d-78c9c88f96-sggpl\" (UID: \"d090174e-7ec3-4c2c-a296-f15f9543ea36\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d7e45d-78c9c88f96-sggpl" Nov 04 01:18:04.774949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.760191 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/507f1fa6-dc41-4285-9e97-941eb8e9f525-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-d7e45d-7ff9858d99-qv8gj\" (UID: \"507f1fa6-dc41-4285-9e97-941eb8e9f525\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d7e45d-7ff9858d99-qv8gj" Nov 04 01:18:04.774949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.760209 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vbj7\" (UniqueName: \"kubernetes.io/projected/d090174e-7ec3-4c2c-a296-f15f9543ea36-kube-api-access-9vbj7\") pod \"payload-processing-e2e-worker-w2-d7e45d-78c9c88f96-sggpl\" (UID: \"d090174e-7ec3-4c2c-a296-f15f9543ea36\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d7e45d-78c9c88f96-sggpl" Nov 04 01:18:04.861507 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.861476 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vbj7\" (UniqueName: \"kubernetes.io/projected/d090174e-7ec3-4c2c-a296-f15f9543ea36-kube-api-access-9vbj7\") pod \"payload-processing-e2e-worker-w2-d7e45d-78c9c88f96-sggpl\" (UID: \"d090174e-7ec3-4c2c-a296-f15f9543ea36\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d7e45d-78c9c88f96-sggpl" Nov 04 01:18:04.861633 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.861589 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwnf9\" (UniqueName: \"kubernetes.io/projected/507f1fa6-dc41-4285-9e97-941eb8e9f525-kube-api-access-nwnf9\") pod \"payload-pre-processing-e2e-worker-w2-d7e45d-7ff9858d99-qv8gj\" (UID: \"507f1fa6-dc41-4285-9e97-941eb8e9f525\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d7e45d-7ff9858d99-qv8gj" Nov 04 01:18:04.861633 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.861624 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d090174e-7ec3-4c2c-a296-f15f9543ea36-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-d7e45d-78c9c88f96-sggpl\" (UID: \"d090174e-7ec3-4c2c-a296-f15f9543ea36\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d7e45d-78c9c88f96-sggpl" Nov 04 01:18:04.861764 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.861693 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/507f1fa6-dc41-4285-9e97-941eb8e9f525-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-d7e45d-7ff9858d99-qv8gj\" (UID: \"507f1fa6-dc41-4285-9e97-941eb8e9f525\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d7e45d-7ff9858d99-qv8gj" Nov 04 01:18:04.862360 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.862342 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/507f1fa6-dc41-4285-9e97-941eb8e9f525-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-d7e45d-7ff9858d99-qv8gj\" (UID: \"507f1fa6-dc41-4285-9e97-941eb8e9f525\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d7e45d-7ff9858d99-qv8gj" Nov 04 01:18:04.862528 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.862500 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d090174e-7ec3-4c2c-a296-f15f9543ea36-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-d7e45d-78c9c88f96-sggpl\" (UID: \"d090174e-7ec3-4c2c-a296-f15f9543ea36\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d7e45d-78c9c88f96-sggpl" Nov 04 01:18:04.869841 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.869816 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vbj7\" (UniqueName: \"kubernetes.io/projected/d090174e-7ec3-4c2c-a296-f15f9543ea36-kube-api-access-9vbj7\") pod \"payload-processing-e2e-worker-w2-d7e45d-78c9c88f96-sggpl\" (UID: \"d090174e-7ec3-4c2c-a296-f15f9543ea36\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d7e45d-78c9c88f96-sggpl" Nov 04 01:18:04.870282 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.870265 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwnf9\" (UniqueName: \"kubernetes.io/projected/507f1fa6-dc41-4285-9e97-941eb8e9f525-kube-api-access-nwnf9\") pod \"payload-pre-processing-e2e-worker-w2-d7e45d-7ff9858d99-qv8gj\" (UID: \"507f1fa6-dc41-4285-9e97-941eb8e9f525\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d7e45d-7ff9858d99-qv8gj" Nov 04 01:18:04.974573 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.974541 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-d7e45d-78c9c88f96-sggpl" Nov 04 01:18:04.998115 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:04.998077 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d7e45d-7ff9858d99-qv8gj" Nov 04 01:18:05.132082 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.132022 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6"] Nov 04 01:18:05.136195 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.136171 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6" Nov 04 01:18:05.145298 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.145274 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-499ddd-serving-cert\"" Nov 04 01:18:05.146467 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.146446 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6"] Nov 04 01:18:05.263529 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.263506 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d7e45d-78c9c88f96-sggpl"] Nov 04 01:18:05.264772 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:18:05.264749 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd090174e_7ec3_4c2c_a296_f15f9543ea36.slice/crio-9597bb7f4eafa2b249841aa5f842c4e1f280c2d58f453ed73c10309350d230f9 WatchSource:0}: Error finding container 9597bb7f4eafa2b249841aa5f842c4e1f280c2d58f453ed73c10309350d230f9: Status 404 returned error can't find the container with id 9597bb7f4eafa2b249841aa5f842c4e1f280c2d58f453ed73c10309350d230f9 Nov 04 01:18:05.269344 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.269313 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqq9k\" (UniqueName: \"kubernetes.io/projected/b9c29ddd-82aa-4855-ba74-fa541b0087d8-kube-api-access-fqq9k\") pod \"maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6\" (UID: \"b9c29ddd-82aa-4855-ba74-fa541b0087d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6" Nov 04 01:18:05.269511 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.269491 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9c29ddd-82aa-4855-ba74-fa541b0087d8-metrics-certs\") pod \"maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6\" (UID: \"b9c29ddd-82aa-4855-ba74-fa541b0087d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6" Nov 04 01:18:05.269570 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.269555 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9c29ddd-82aa-4855-ba74-fa541b0087d8-maas-api-tls\") pod \"maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6\" (UID: \"b9c29ddd-82aa-4855-ba74-fa541b0087d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6" Nov 04 01:18:05.269672 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.269655 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9c29ddd-82aa-4855-ba74-fa541b0087d8-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6\" (UID: \"b9c29ddd-82aa-4855-ba74-fa541b0087d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6" Nov 04 01:18:05.293040 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.293016 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ee876a-54f4fd798c-djxkh"] Nov 04 01:18:05.298435 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.298412 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee876a-54f4fd798c-djxkh" Nov 04 01:18:05.299826 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.299801 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d7e45d-7ff9858d99-qv8gj"] Nov 04 01:18:05.313043 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.308633 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ee876a-54f4fd798c-djxkh"] Nov 04 01:18:05.313043 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.310043 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-ee876a-dockercfg-9kjqt\"" Nov 04 01:18:05.313043 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.310113 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-ee876a\"" Nov 04 01:18:05.370597 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.370547 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqq9k\" (UniqueName: \"kubernetes.io/projected/b9c29ddd-82aa-4855-ba74-fa541b0087d8-kube-api-access-fqq9k\") pod \"maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6\" (UID: \"b9c29ddd-82aa-4855-ba74-fa541b0087d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6" Nov 04 01:18:05.370705 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.370659 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a70660e3-f657-4f28-9743-651390c8ae3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ee876a-54f4fd798c-djxkh\" (UID: \"a70660e3-f657-4f28-9743-651390c8ae3f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee876a-54f4fd798c-djxkh" Nov 04 01:18:05.370767 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.370700 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bwrx\" (UniqueName: \"kubernetes.io/projected/a70660e3-f657-4f28-9743-651390c8ae3f-kube-api-access-4bwrx\") pod \"payload-pre-processing-e2e-worker-w0-ee876a-54f4fd798c-djxkh\" (UID: \"a70660e3-f657-4f28-9743-651390c8ae3f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee876a-54f4fd798c-djxkh" Nov 04 01:18:05.370828 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.370767 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9c29ddd-82aa-4855-ba74-fa541b0087d8-metrics-certs\") pod \"maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6\" (UID: \"b9c29ddd-82aa-4855-ba74-fa541b0087d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6" Nov 04 01:18:05.370828 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.370814 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9c29ddd-82aa-4855-ba74-fa541b0087d8-maas-api-tls\") pod \"maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6\" (UID: \"b9c29ddd-82aa-4855-ba74-fa541b0087d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6" Nov 04 01:18:05.370967 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.370889 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9c29ddd-82aa-4855-ba74-fa541b0087d8-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6\" (UID: \"b9c29ddd-82aa-4855-ba74-fa541b0087d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6" Nov 04 01:18:05.371667 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.371640 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9c29ddd-82aa-4855-ba74-fa541b0087d8-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6\" (UID: \"b9c29ddd-82aa-4855-ba74-fa541b0087d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6" Nov 04 01:18:05.375053 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.375011 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9c29ddd-82aa-4855-ba74-fa541b0087d8-maas-api-tls\") pod \"maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6\" (UID: \"b9c29ddd-82aa-4855-ba74-fa541b0087d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6" Nov 04 01:18:05.376121 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.376020 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-499ddd-68c97cbdc5-57gdf" event={"ID":"1510cc32-e22b-46f0-acfb-3e2994767889","Type":"ContainerStarted","Data":"b9a842d44d3a0eaa9b17b2883ccb01baaf9bd760bb3cf3f40bd2c6823478ec75"} Nov 04 01:18:05.376121 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.376081 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-499ddd-68c97cbdc5-57gdf" event={"ID":"1510cc32-e22b-46f0-acfb-3e2994767889","Type":"ContainerStarted","Data":"5fe928c771a4d888b4a000b342e004349e19cd720f5db91dd8cc9b4ad3b9417e"} Nov 04 01:18:05.376851 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.376796 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-499ddd-68c97cbdc5-57gdf" Nov 04 01:18:05.378952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.378574 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn" event={"ID":"facc3749-8125-4daa-956e-e521856cc7da","Type":"ContainerStarted","Data":"48022948fea01455582e00181bbb3a8a312d106a85f98321c7e39e4d7a06846d"} Nov 04 01:18:05.378952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.378622 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn" event={"ID":"facc3749-8125-4daa-956e-e521856cc7da","Type":"ContainerStarted","Data":"37ab04886888dab059a554a1163e581285571810525ebb3b10ac33ae6c016fae"} Nov 04 01:18:05.379979 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.379461 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn" Nov 04 01:18:05.380733 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.380549 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqq9k\" (UniqueName: \"kubernetes.io/projected/b9c29ddd-82aa-4855-ba74-fa541b0087d8-kube-api-access-fqq9k\") pod \"maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6\" (UID: \"b9c29ddd-82aa-4855-ba74-fa541b0087d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6" Nov 04 01:18:05.381009 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.380819 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9c29ddd-82aa-4855-ba74-fa541b0087d8-metrics-certs\") pod \"maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6\" (UID: \"b9c29ddd-82aa-4855-ba74-fa541b0087d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6" Nov 04 01:18:05.382251 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.382087 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d7e45d-7ff9858d99-qv8gj" event={"ID":"507f1fa6-dc41-4285-9e97-941eb8e9f525","Type":"ContainerStarted","Data":"5285cea581c50ef2e87c4767c1f3f66d5b736baf4a1105589e8c7dae0760f8cb"} Nov 04 01:18:05.383502 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.383486 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-d7e45d-78c9c88f96-sggpl" event={"ID":"d090174e-7ec3-4c2c-a296-f15f9543ea36","Type":"ContainerStarted","Data":"9597bb7f4eafa2b249841aa5f842c4e1f280c2d58f453ed73c10309350d230f9"} Nov 04 01:18:05.384707 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.384694 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-d7e45d-78c9c88f96-sggpl" Nov 04 01:18:05.387500 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.386257 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-499ddd-677fd64986-9c76n" event={"ID":"22a00871-73e9-4c9a-aaa6-9046b1004b9e","Type":"ContainerStarted","Data":"d95133a3271c83c30f652fddc6e5f2669944483122712d26d91d65a6e74d87bd"} Nov 04 01:18:05.387500 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.386289 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-499ddd-677fd64986-9c76n" Nov 04 01:18:05.387500 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.386304 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-499ddd-677fd64986-9c76n" event={"ID":"22a00871-73e9-4c9a-aaa6-9046b1004b9e","Type":"ContainerStarted","Data":"77a4bc64fbd8dd01eeff26cbea3a7d518513f9b48baa0eeeb4bc111f082df62e"} Nov 04 01:18:05.405481 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.404854 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-499ddd-68c97cbdc5-57gdf" podStartSLOduration=1.404837683 podStartE2EDuration="1.404837683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:05.402106171 +0000 UTC m=+1170.343955097" watchObservedRunningTime="2026-11-04 01:18:05.404837683 +0000 UTC m=+1170.346686586" Nov 04 01:18:05.421598 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.421544 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn" podStartSLOduration=2.038351023 podStartE2EDuration="2.421527429s" totalImagesPullingTime="383.176406ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:05.41918595 +0000 UTC m=+1170.361034882" watchObservedRunningTime="2026-11-04 01:18:05.421527429 +0000 UTC m=+1170.363376333" Nov 04 01:18:05.448863 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.441511 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-d7e45d-78c9c88f96-sggpl" podStartSLOduration=1.441495249 podStartE2EDuration="1.441495249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:05.439124404 +0000 UTC m=+1170.380973312" watchObservedRunningTime="2026-11-04 01:18:05.441495249 +0000 UTC m=+1170.383344153" Nov 04 01:18:05.456165 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.456135 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6" Nov 04 01:18:05.468214 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.468168 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-499ddd-677fd64986-9c76n" podStartSLOduration=1.468152387 podStartE2EDuration="1.468152387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:05.467261345 +0000 UTC m=+1170.409110249" watchObservedRunningTime="2026-11-04 01:18:05.468152387 +0000 UTC m=+1170.410001290" Nov 04 01:18:05.472833 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.472803 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a70660e3-f657-4f28-9743-651390c8ae3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ee876a-54f4fd798c-djxkh\" (UID: \"a70660e3-f657-4f28-9743-651390c8ae3f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee876a-54f4fd798c-djxkh" Nov 04 01:18:05.472989 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.472845 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bwrx\" (UniqueName: \"kubernetes.io/projected/a70660e3-f657-4f28-9743-651390c8ae3f-kube-api-access-4bwrx\") pod \"payload-pre-processing-e2e-worker-w0-ee876a-54f4fd798c-djxkh\" (UID: \"a70660e3-f657-4f28-9743-651390c8ae3f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee876a-54f4fd798c-djxkh" Nov 04 01:18:05.476670 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.476600 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a70660e3-f657-4f28-9743-651390c8ae3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ee876a-54f4fd798c-djxkh\" (UID: \"a70660e3-f657-4f28-9743-651390c8ae3f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee876a-54f4fd798c-djxkh" Nov 04 01:18:05.488794 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.488770 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bwrx\" (UniqueName: \"kubernetes.io/projected/a70660e3-f657-4f28-9743-651390c8ae3f-kube-api-access-4bwrx\") pod \"payload-pre-processing-e2e-worker-w0-ee876a-54f4fd798c-djxkh\" (UID: \"a70660e3-f657-4f28-9743-651390c8ae3f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee876a-54f4fd798c-djxkh" Nov 04 01:18:05.632543 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.632512 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee876a-54f4fd798c-djxkh" Nov 04 01:18:05.660665 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.660635 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-586541-6bdd6489cf-pn6dk"] Nov 04 01:18:05.666672 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.666643 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-586541-6bdd6489cf-pn6dk" Nov 04 01:18:05.673415 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.673389 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-586541-dockercfg-q48tz\"" Nov 04 01:18:05.680467 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.680443 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-586541-6bdd6489cf-pn6dk"] Nov 04 01:18:05.754530 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:18:05.754501 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c29ddd_82aa_4855_ba74_fa541b0087d8.slice/crio-b786b0ec118c43f54120284bc10d764ff6603b0d1f391641c030d0f4f22b5148 WatchSource:0}: Error finding container b786b0ec118c43f54120284bc10d764ff6603b0d1f391641c030d0f4f22b5148: Status 404 returned error can't find the container with id b786b0ec118c43f54120284bc10d764ff6603b0d1f391641c030d0f4f22b5148 Nov 04 01:18:05.755319 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.755300 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-586541\"" Nov 04 01:18:05.761715 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.761688 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6"] Nov 04 01:18:05.781709 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.781649 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgbmg\" (UniqueName: \"kubernetes.io/projected/b3ea0000-873e-4789-8ad1-d182a29d4489-kube-api-access-sgbmg\") pod \"payload-pre-processing-e2e-worker-w1-586541-6bdd6489cf-pn6dk\" (UID: \"b3ea0000-873e-4789-8ad1-d182a29d4489\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-586541-6bdd6489cf-pn6dk" Nov 04 01:18:05.781806 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.781712 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ea0000-873e-4789-8ad1-d182a29d4489-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-586541-6bdd6489cf-pn6dk\" (UID: \"b3ea0000-873e-4789-8ad1-d182a29d4489\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-586541-6bdd6489cf-pn6dk" Nov 04 01:18:05.882718 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.882673 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ea0000-873e-4789-8ad1-d182a29d4489-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-586541-6bdd6489cf-pn6dk\" (UID: \"b3ea0000-873e-4789-8ad1-d182a29d4489\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-586541-6bdd6489cf-pn6dk" Nov 04 01:18:05.882895 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.882876 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgbmg\" (UniqueName: \"kubernetes.io/projected/b3ea0000-873e-4789-8ad1-d182a29d4489-kube-api-access-sgbmg\") pod \"payload-pre-processing-e2e-worker-w1-586541-6bdd6489cf-pn6dk\" (UID: \"b3ea0000-873e-4789-8ad1-d182a29d4489\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-586541-6bdd6489cf-pn6dk" Nov 04 01:18:05.883736 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.883708 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ea0000-873e-4789-8ad1-d182a29d4489-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-586541-6bdd6489cf-pn6dk\" (UID: \"b3ea0000-873e-4789-8ad1-d182a29d4489\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-586541-6bdd6489cf-pn6dk" Nov 04 01:18:05.891744 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.891722 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgbmg\" (UniqueName: \"kubernetes.io/projected/b3ea0000-873e-4789-8ad1-d182a29d4489-kube-api-access-sgbmg\") pod \"payload-pre-processing-e2e-worker-w1-586541-6bdd6489cf-pn6dk\" (UID: \"b3ea0000-873e-4789-8ad1-d182a29d4489\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-586541-6bdd6489cf-pn6dk" Nov 04 01:18:05.926280 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.926252 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ee876a-54f4fd798c-djxkh"] Nov 04 01:18:05.928948 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:18:05.928909 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70660e3_f657_4f28_9743_651390c8ae3f.slice/crio-b2bdd8787b27ee3afd9f49043bba23c71d86c095705a5cdb7eb0029aad5d0efc WatchSource:0}: Error finding container b2bdd8787b27ee3afd9f49043bba23c71d86c095705a5cdb7eb0029aad5d0efc: Status 404 returned error can't find the container with id b2bdd8787b27ee3afd9f49043bba23c71d86c095705a5cdb7eb0029aad5d0efc Nov 04 01:18:05.991820 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:05.991794 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-586541-6bdd6489cf-pn6dk" Nov 04 01:18:06.032226 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.032153 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6"] Nov 04 01:18:06.036884 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.036857 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6" Nov 04 01:18:06.049903 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.049881 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6"] Nov 04 01:18:06.050311 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.050290 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-d7e45d-serving-cert\"" Nov 04 01:18:06.186463 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.186436 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4016d397-9682-4387-b042-68e1ef09f58b-maas-api-tls\") pod \"maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6\" (UID: \"4016d397-9682-4387-b042-68e1ef09f58b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6" Nov 04 01:18:06.186589 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.186572 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4016d397-9682-4387-b042-68e1ef09f58b-metrics-certs\") pod \"maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6\" (UID: \"4016d397-9682-4387-b042-68e1ef09f58b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6" Nov 04 01:18:06.186634 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.186606 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgwrd\" (UniqueName: \"kubernetes.io/projected/4016d397-9682-4387-b042-68e1ef09f58b-kube-api-access-fgwrd\") pod \"maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6\" (UID: \"4016d397-9682-4387-b042-68e1ef09f58b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6" Nov 04 01:18:06.186679 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.186635 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4016d397-9682-4387-b042-68e1ef09f58b-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6\" (UID: \"4016d397-9682-4387-b042-68e1ef09f58b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6" Nov 04 01:18:06.283327 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.283300 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-586541-6bdd6489cf-pn6dk"] Nov 04 01:18:06.285570 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:18:06.285539 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ea0000_873e_4789_8ad1_d182a29d4489.slice/crio-8906446ddcc2ae0407d9527acca1ee748764b03882e2022a89c08c728c1b016b WatchSource:0}: Error finding container 8906446ddcc2ae0407d9527acca1ee748764b03882e2022a89c08c728c1b016b: Status 404 returned error can't find the container with id 8906446ddcc2ae0407d9527acca1ee748764b03882e2022a89c08c728c1b016b Nov 04 01:18:06.287401 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.287377 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4016d397-9682-4387-b042-68e1ef09f58b-metrics-certs\") pod \"maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6\" (UID: \"4016d397-9682-4387-b042-68e1ef09f58b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6" Nov 04 01:18:06.287488 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.287413 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgwrd\" (UniqueName: \"kubernetes.io/projected/4016d397-9682-4387-b042-68e1ef09f58b-kube-api-access-fgwrd\") pod \"maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6\" (UID: \"4016d397-9682-4387-b042-68e1ef09f58b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6" Nov 04 01:18:06.287488 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.287442 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4016d397-9682-4387-b042-68e1ef09f58b-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6\" (UID: \"4016d397-9682-4387-b042-68e1ef09f58b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6" Nov 04 01:18:06.287594 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.287578 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4016d397-9682-4387-b042-68e1ef09f58b-maas-api-tls\") pod \"maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6\" (UID: \"4016d397-9682-4387-b042-68e1ef09f58b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6" Nov 04 01:18:06.288210 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.288189 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4016d397-9682-4387-b042-68e1ef09f58b-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6\" (UID: \"4016d397-9682-4387-b042-68e1ef09f58b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6" Nov 04 01:18:06.290027 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.290007 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4016d397-9682-4387-b042-68e1ef09f58b-metrics-certs\") pod \"maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6\" (UID: \"4016d397-9682-4387-b042-68e1ef09f58b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6" Nov 04 01:18:06.290224 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.290168 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4016d397-9682-4387-b042-68e1ef09f58b-maas-api-tls\") pod \"maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6\" (UID: \"4016d397-9682-4387-b042-68e1ef09f58b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6" Nov 04 01:18:06.300951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.300913 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgwrd\" (UniqueName: \"kubernetes.io/projected/4016d397-9682-4387-b042-68e1ef09f58b-kube-api-access-fgwrd\") pod \"maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6\" (UID: \"4016d397-9682-4387-b042-68e1ef09f58b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6" Nov 04 01:18:06.366107 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.366079 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6" Nov 04 01:18:06.395679 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.395647 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee876a-54f4fd798c-djxkh" event={"ID":"a70660e3-f657-4f28-9743-651390c8ae3f","Type":"ContainerStarted","Data":"1859918638a2b822c70df32a32e0fcc5e1ad34c9d4f31488d28724f7c051a4ed"} Nov 04 01:18:06.395807 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.395728 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee876a-54f4fd798c-djxkh" event={"ID":"a70660e3-f657-4f28-9743-651390c8ae3f","Type":"ContainerStarted","Data":"b2bdd8787b27ee3afd9f49043bba23c71d86c095705a5cdb7eb0029aad5d0efc"} Nov 04 01:18:06.395888 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.395868 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee876a-54f4fd798c-djxkh" Nov 04 01:18:06.397527 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.397491 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-586541-6bdd6489cf-pn6dk" event={"ID":"b3ea0000-873e-4789-8ad1-d182a29d4489","Type":"ContainerStarted","Data":"5d8b7ecc7b4fdc5d5334c45804632a8719cc77ea8c3bf203baa05165cc5a6f76"} Nov 04 01:18:06.397527 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.397544 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-586541-6bdd6489cf-pn6dk" Nov 04 01:18:06.397803 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.397572 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-586541-6bdd6489cf-pn6dk" event={"ID":"b3ea0000-873e-4789-8ad1-d182a29d4489","Type":"ContainerStarted","Data":"8906446ddcc2ae0407d9527acca1ee748764b03882e2022a89c08c728c1b016b"} Nov 04 01:18:06.399686 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.399660 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d7e45d-7ff9858d99-qv8gj" event={"ID":"507f1fa6-dc41-4285-9e97-941eb8e9f525","Type":"ContainerStarted","Data":"b53b2c978346dd9a897ea0aa2ac729d2e5a0924eff3c68090e0b573a5cd6e554"} Nov 04 01:18:06.399846 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.399805 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d7e45d-7ff9858d99-qv8gj" Nov 04 01:18:06.402885 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.402863 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-d7e45d-78c9c88f96-sggpl" event={"ID":"d090174e-7ec3-4c2c-a296-f15f9543ea36","Type":"ContainerStarted","Data":"bc649ae8fe26ee698d8c31f856e194fd2da184ae32004b377d05bc5cf06bd468"} Nov 04 01:18:06.405818 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.404802 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6" event={"ID":"b9c29ddd-82aa-4855-ba74-fa541b0087d8","Type":"ContainerStarted","Data":"619ea51e3ee310ce0c365fead6098790c703811ccd4ad03253cde74cda180a5a"} Nov 04 01:18:06.405818 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.404849 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6" event={"ID":"b9c29ddd-82aa-4855-ba74-fa541b0087d8","Type":"ContainerStarted","Data":"b786b0ec118c43f54120284bc10d764ff6603b0d1f391641c030d0f4f22b5148"} Nov 04 01:18:06.405818 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.405548 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6" Nov 04 01:18:06.422097 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.422035 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee876a-54f4fd798c-djxkh" podStartSLOduration=1.422019286 podStartE2EDuration="1.422019286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:06.421087945 +0000 UTC m=+1171.362936859" watchObservedRunningTime="2026-11-04 01:18:06.422019286 +0000 UTC m=+1171.363868190" Nov 04 01:18:06.453787 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.453701 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6" podStartSLOduration=1.049541646 podStartE2EDuration="1.453684018s" totalImagesPullingTime="404.142372ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:06.451909173 +0000 UTC m=+1171.393758077" watchObservedRunningTime="2026-11-04 01:18:06.453684018 +0000 UTC m=+1171.395532926" Nov 04 01:18:06.470723 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.470674 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d7e45d-7ff9858d99-qv8gj" podStartSLOduration=2.470659832 podStartE2EDuration="2.470659832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:06.467842734 +0000 UTC m=+1171.409691637" watchObservedRunningTime="2026-11-04 01:18:06.470659832 +0000 UTC m=+1171.412508712" Nov 04 01:18:06.490282 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:18:06.490162 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:18:06.493610 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.493558 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-586541-6bdd6489cf-pn6dk" podStartSLOduration=1.493540956 podStartE2EDuration="1.493540956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:06.490566879 +0000 UTC m=+1171.432415779" watchObservedRunningTime="2026-11-04 01:18:06.493540956 +0000 UTC m=+1171.435389864" Nov 04 01:18:06.658044 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:06.658015 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6"] Nov 04 01:18:06.659274 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:18:06.659250 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4016d397_9682_4387_b042_68e1ef09f58b.slice/crio-8827b2d79245de054f6d356226acb83976f2f8cd4651f1d90ffdb94aac165936 WatchSource:0}: Error finding container 8827b2d79245de054f6d356226acb83976f2f8cd4651f1d90ffdb94aac165936: Status 404 returned error can't find the container with id 8827b2d79245de054f6d356226acb83976f2f8cd4651f1d90ffdb94aac165936 Nov 04 01:18:07.413886 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:07.413845 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6" event={"ID":"4016d397-9682-4387-b042-68e1ef09f58b","Type":"ContainerStarted","Data":"c736b1ddc66f9d44e5a2d5ced7fc3fc748651ab9b43cac148dc8630a2c52e0f8"} Nov 04 01:18:07.414335 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:07.413918 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6" event={"ID":"4016d397-9682-4387-b042-68e1ef09f58b","Type":"ContainerStarted","Data":"8827b2d79245de054f6d356226acb83976f2f8cd4651f1d90ffdb94aac165936"} Nov 04 01:18:07.414335 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:07.414136 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6" Nov 04 01:18:07.447832 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:07.447741 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6" podStartSLOduration=0.928572078 podStartE2EDuration="1.447727501s" totalImagesPullingTime="519.155423ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:07.446002498 +0000 UTC m=+1172.387851414" watchObservedRunningTime="2026-11-04 01:18:07.447727501 +0000 UTC m=+1172.389576403" Nov 04 01:18:08.596734 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:08.596697 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf"] Nov 04 01:18:08.602454 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:08.602429 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf" Nov 04 01:18:08.615712 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:08.612270 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-586541-serving-cert\"" Nov 04 01:18:08.615712 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:08.615062 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf"] Nov 04 01:18:08.717431 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:08.717400 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d75e9829-735d-4f1c-b79b-55b46d5cfc01-maas-api-tls\") pod \"maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf\" (UID: \"d75e9829-735d-4f1c-b79b-55b46d5cfc01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf" Nov 04 01:18:08.717587 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:08.717466 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d75e9829-735d-4f1c-b79b-55b46d5cfc01-metrics-certs\") pod \"maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf\" (UID: \"d75e9829-735d-4f1c-b79b-55b46d5cfc01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf" Nov 04 01:18:08.717666 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:08.717645 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d75e9829-735d-4f1c-b79b-55b46d5cfc01-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf\" (UID: \"d75e9829-735d-4f1c-b79b-55b46d5cfc01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf" Nov 04 01:18:08.717707 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:08.717692 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62pkl\" (UniqueName: \"kubernetes.io/projected/d75e9829-735d-4f1c-b79b-55b46d5cfc01-kube-api-access-62pkl\") pod \"maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf\" (UID: \"d75e9829-735d-4f1c-b79b-55b46d5cfc01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf" Nov 04 01:18:08.830518 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:08.830488 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62pkl\" (UniqueName: \"kubernetes.io/projected/d75e9829-735d-4f1c-b79b-55b46d5cfc01-kube-api-access-62pkl\") pod \"maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf\" (UID: \"d75e9829-735d-4f1c-b79b-55b46d5cfc01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf" Nov 04 01:18:08.830719 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:08.830529 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d75e9829-735d-4f1c-b79b-55b46d5cfc01-maas-api-tls\") pod \"maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf\" (UID: \"d75e9829-735d-4f1c-b79b-55b46d5cfc01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf" Nov 04 01:18:08.830719 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:08.830607 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d75e9829-735d-4f1c-b79b-55b46d5cfc01-metrics-certs\") pod \"maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf\" (UID: \"d75e9829-735d-4f1c-b79b-55b46d5cfc01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf" Nov 04 01:18:08.830862 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:08.830752 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d75e9829-735d-4f1c-b79b-55b46d5cfc01-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf\" (UID: \"d75e9829-735d-4f1c-b79b-55b46d5cfc01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf" Nov 04 01:18:08.831951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:08.831683 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d75e9829-735d-4f1c-b79b-55b46d5cfc01-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf\" (UID: \"d75e9829-735d-4f1c-b79b-55b46d5cfc01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf" Nov 04 01:18:08.835945 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:08.833842 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d75e9829-735d-4f1c-b79b-55b46d5cfc01-metrics-certs\") pod \"maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf\" (UID: \"d75e9829-735d-4f1c-b79b-55b46d5cfc01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf" Nov 04 01:18:08.835945 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:08.834034 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d75e9829-735d-4f1c-b79b-55b46d5cfc01-maas-api-tls\") pod \"maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf\" (UID: \"d75e9829-735d-4f1c-b79b-55b46d5cfc01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf" Nov 04 01:18:08.841882 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:08.841853 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62pkl\" (UniqueName: \"kubernetes.io/projected/d75e9829-735d-4f1c-b79b-55b46d5cfc01-kube-api-access-62pkl\") pod \"maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf\" (UID: \"d75e9829-735d-4f1c-b79b-55b46d5cfc01\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf" Nov 04 01:18:08.926396 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:08.926369 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf" Nov 04 01:18:09.205898 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:09.205865 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf"] Nov 04 01:18:09.206735 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:18:09.206705 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd75e9829_735d_4f1c_b79b_55b46d5cfc01.slice/crio-5da7d8541c190fcbd5d0063c887c514f398b264b7b489f979bbc957f42a99129 WatchSource:0}: Error finding container 5da7d8541c190fcbd5d0063c887c514f398b264b7b489f979bbc957f42a99129: Status 404 returned error can't find the container with id 5da7d8541c190fcbd5d0063c887c514f398b264b7b489f979bbc957f42a99129 Nov 04 01:18:09.439322 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:09.439285 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf" event={"ID":"d75e9829-735d-4f1c-b79b-55b46d5cfc01","Type":"ContainerStarted","Data":"5da7d8541c190fcbd5d0063c887c514f398b264b7b489f979bbc957f42a99129"} Nov 04 01:18:10.393364 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:10.393333 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb" Nov 04 01:18:10.452205 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:10.452155 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf" event={"ID":"d75e9829-735d-4f1c-b79b-55b46d5cfc01","Type":"ContainerStarted","Data":"cee0774be47adcf1df88dc4f33e9431833777e7f9d354c320d04457b8f3bc489"} Nov 04 01:18:10.452402 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:10.452235 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf" Nov 04 01:18:10.474175 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:10.474129 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf" podStartSLOduration=2.007606596 podStartE2EDuration="2.474116605s" totalImagesPullingTime="466.510009ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:10.471685946 +0000 UTC m=+1175.413534851" watchObservedRunningTime="2026-11-04 01:18:10.474116605 +0000 UTC m=+1175.415965533" Nov 04 01:18:12.419981 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:12.419952 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn" Nov 04 01:18:13.432228 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:13.432196 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6" Nov 04 01:18:13.432729 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:13.432285 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6" Nov 04 01:18:14.344642 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:14.344612 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-9c8bf5-646c6d9774-4862q" Nov 04 01:18:14.345338 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:14.345311 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9c8bf5-9758b5cf-9hsn2" Nov 04 01:18:15.389688 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:15.389658 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-ca55c1-66d7c55b44-mm9qn" Nov 04 01:18:15.390227 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:15.389744 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ca55c1-5bb65b58bc-j9bkv" Nov 04 01:18:15.390387 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:15.390351 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-099dd9-775897687b-7z7mn" Nov 04 01:18:16.408530 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:16.408488 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-099dd9-5f4444cd6c-g7kl6" Nov 04 01:18:16.408961 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:16.408670 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-499ddd-68c97cbdc5-57gdf" Nov 04 01:18:16.408961 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:16.408710 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-499ddd-677fd64986-9c76n" Nov 04 01:18:16.466623 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:16.466593 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf" Nov 04 01:18:17.417782 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:17.417738 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-586541-6bdd6489cf-pn6dk" Nov 04 01:18:17.418307 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:17.418282 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-d7e45d-78c9c88f96-sggpl" Nov 04 01:18:17.418437 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:17.418342 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee876a-54f4fd798c-djxkh" Nov 04 01:18:17.418437 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:17.418378 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d7e45d-7ff9858d99-qv8gj" Nov 04 01:18:36.481088 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:18:36.481053 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:18:38.072572 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:38.072533 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z"] Nov 04 01:18:38.078613 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:38.078593 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:18:38.084576 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:38.084552 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-586541\"/\"kube-root-ca.crt\"" Nov 04 01:18:38.084576 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:38.084552 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-586541\"/\"openshift-service-ca.crt\"" Nov 04 01:18:38.085194 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:38.085176 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-586541\"/\"default-dockercfg-26p4v\"" Nov 04 01:18:38.085339 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:38.085247 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-586541\"/\"simulator-w1-586541-kserve-self-signed-certs\"" Nov 04 01:18:38.086379 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:38.086356 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z"] Nov 04 01:18:38.245361 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:38.245322 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b51f24e-65d2-40fd-8b78-70ee1a593d00-tmp-dir\") pod \"simulator-w1-586541-kserve-6fc9f9dcb5-52w9z\" (UID: \"1b51f24e-65d2-40fd-8b78-70ee1a593d00\") " pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:18:38.245540 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:38.245435 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b51f24e-65d2-40fd-8b78-70ee1a593d00-home\") pod \"simulator-w1-586541-kserve-6fc9f9dcb5-52w9z\" (UID: \"1b51f24e-65d2-40fd-8b78-70ee1a593d00\") " pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:18:38.245540 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:38.245474 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b51f24e-65d2-40fd-8b78-70ee1a593d00-tls-certs\") pod \"simulator-w1-586541-kserve-6fc9f9dcb5-52w9z\" (UID: \"1b51f24e-65d2-40fd-8b78-70ee1a593d00\") " pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:18:38.245617 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:38.245590 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b51f24e-65d2-40fd-8b78-70ee1a593d00-model-cache\") pod \"simulator-w1-586541-kserve-6fc9f9dcb5-52w9z\" (UID: \"1b51f24e-65d2-40fd-8b78-70ee1a593d00\") " pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:18:38.245651 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:38.245622 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whbzr\" (UniqueName: \"kubernetes.io/projected/1b51f24e-65d2-40fd-8b78-70ee1a593d00-kube-api-access-whbzr\") pod \"simulator-w1-586541-kserve-6fc9f9dcb5-52w9z\" (UID: \"1b51f24e-65d2-40fd-8b78-70ee1a593d00\") " pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:18:38.245688 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:38.245658 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b51f24e-65d2-40fd-8b78-70ee1a593d00-dshm\") pod \"simulator-w1-586541-kserve-6fc9f9dcb5-52w9z\" (UID: \"1b51f24e-65d2-40fd-8b78-70ee1a593d00\") " pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:18:38.346992 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:38.346886 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whbzr\" (UniqueName: \"kubernetes.io/projected/1b51f24e-65d2-40fd-8b78-70ee1a593d00-kube-api-access-whbzr\") pod \"simulator-w1-586541-kserve-6fc9f9dcb5-52w9z\" (UID: \"1b51f24e-65d2-40fd-8b78-70ee1a593d00\") " pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:18:38.346992 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:38.346955 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b51f24e-65d2-40fd-8b78-70ee1a593d00-dshm\") pod \"simulator-w1-586541-kserve-6fc9f9dcb5-52w9z\" (UID: \"1b51f24e-65d2-40fd-8b78-70ee1a593d00\") " pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:18:38.347216 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:38.347034 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b51f24e-65d2-40fd-8b78-70ee1a593d00-tmp-dir\") pod \"simulator-w1-586541-kserve-6fc9f9dcb5-52w9z\" (UID: \"1b51f24e-65d2-40fd-8b78-70ee1a593d00\") " pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:18:38.347216 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:38.347084 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b51f24e-65d2-40fd-8b78-70ee1a593d00-home\") pod \"simulator-w1-586541-kserve-6fc9f9dcb5-52w9z\" (UID: \"1b51f24e-65d2-40fd-8b78-70ee1a593d00\") " pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:18:38.347216 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:38.347115 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b51f24e-65d2-40fd-8b78-70ee1a593d00-tls-certs\") pod \"simulator-w1-586541-kserve-6fc9f9dcb5-52w9z\" (UID: \"1b51f24e-65d2-40fd-8b78-70ee1a593d00\") " pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:18:38.347216 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:38.347176 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b51f24e-65d2-40fd-8b78-70ee1a593d00-model-cache\") pod \"simulator-w1-586541-kserve-6fc9f9dcb5-52w9z\" (UID: \"1b51f24e-65d2-40fd-8b78-70ee1a593d00\") " pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:18:38.347528 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:38.347502 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b51f24e-65d2-40fd-8b78-70ee1a593d00-home\") pod \"simulator-w1-586541-kserve-6fc9f9dcb5-52w9z\" (UID: \"1b51f24e-65d2-40fd-8b78-70ee1a593d00\") " pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:18:38.347582 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:38.347513 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b51f24e-65d2-40fd-8b78-70ee1a593d00-tmp-dir\") pod \"simulator-w1-586541-kserve-6fc9f9dcb5-52w9z\" (UID: \"1b51f24e-65d2-40fd-8b78-70ee1a593d00\") " pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:18:38.347582 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:38.347563 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b51f24e-65d2-40fd-8b78-70ee1a593d00-model-cache\") pod \"simulator-w1-586541-kserve-6fc9f9dcb5-52w9z\" (UID: \"1b51f24e-65d2-40fd-8b78-70ee1a593d00\") " pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:18:38.349516 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:38.349483 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b51f24e-65d2-40fd-8b78-70ee1a593d00-dshm\") pod \"simulator-w1-586541-kserve-6fc9f9dcb5-52w9z\" (UID: \"1b51f24e-65d2-40fd-8b78-70ee1a593d00\") " pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:18:38.349796 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:38.349776 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b51f24e-65d2-40fd-8b78-70ee1a593d00-tls-certs\") pod \"simulator-w1-586541-kserve-6fc9f9dcb5-52w9z\" (UID: \"1b51f24e-65d2-40fd-8b78-70ee1a593d00\") " pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:18:38.356811 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:38.356786 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whbzr\" (UniqueName: \"kubernetes.io/projected/1b51f24e-65d2-40fd-8b78-70ee1a593d00-kube-api-access-whbzr\") pod \"simulator-w1-586541-kserve-6fc9f9dcb5-52w9z\" (UID: \"1b51f24e-65d2-40fd-8b78-70ee1a593d00\") " pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:18:38.393328 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:38.393300 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:18:38.674549 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:38.674524 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z"] Nov 04 01:18:38.676411 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:18:38.676384 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b51f24e_65d2_40fd_8b78_70ee1a593d00.slice/crio-9904beb4a3242c05ba8f8e738870c428d549bde1913556f92925d8ce7b7570f4 WatchSource:0}: Error finding container 9904beb4a3242c05ba8f8e738870c428d549bde1913556f92925d8ce7b7570f4: Status 404 returned error can't find the container with id 9904beb4a3242c05ba8f8e738870c428d549bde1913556f92925d8ce7b7570f4 Nov 04 01:18:39.649918 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:39.649873 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" event={"ID":"1b51f24e-65d2-40fd-8b78-70ee1a593d00","Type":"ContainerStarted","Data":"49cb68ad09d1bf01650b6c0f82ecdd96bbd139e9032541b673d1da51560c5a09"} Nov 04 01:18:39.650346 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:39.649953 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" event={"ID":"1b51f24e-65d2-40fd-8b78-70ee1a593d00","Type":"ContainerStarted","Data":"9904beb4a3242c05ba8f8e738870c428d549bde1913556f92925d8ce7b7570f4"} Nov 04 01:18:39.668362 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:39.668318 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podStartSLOduration=1.668305554 podStartE2EDuration="1.668305554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:39.666225939 +0000 UTC m=+1204.608074873" watchObservedRunningTime="2026-11-04 01:18:39.668305554 +0000 UTC m=+1204.610154456" Nov 04 01:18:40.393804 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:40.393775 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:18:48.393590 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:48.393538 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:18:48.406575 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:48.406551 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:18:48.718114 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:18:48.718087 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:19:04.777320 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:04.777287 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp"] Nov 04 01:19:04.781844 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:04.781817 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:19:04.788724 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:04.788700 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ee876a\"/\"openshift-service-ca.crt\"" Nov 04 01:19:04.789433 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:04.789407 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp"] Nov 04 01:19:04.790510 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:04.790489 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ee876a\"/\"kube-root-ca.crt\"" Nov 04 01:19:04.790698 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:04.790682 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ee876a\"/\"premium-w0-ee876a-kserve-self-signed-certs\"" Nov 04 01:19:04.791711 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:04.791688 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ee876a\"/\"default-dockercfg-dhtjx\"" Nov 04 01:19:04.936820 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:04.936784 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xf5m\" (UniqueName: \"kubernetes.io/projected/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-kube-api-access-9xf5m\") pod \"premium-w0-ee876a-kserve-67b45d4948-n6tcp\" (UID: \"8b3ccd4d-549f-4621-908f-5be7a1e2a99a\") " pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:19:04.937038 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:04.936963 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-home\") pod \"premium-w0-ee876a-kserve-67b45d4948-n6tcp\" (UID: \"8b3ccd4d-549f-4621-908f-5be7a1e2a99a\") " pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:19:04.937038 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:04.937013 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-tls-certs\") pod \"premium-w0-ee876a-kserve-67b45d4948-n6tcp\" (UID: \"8b3ccd4d-549f-4621-908f-5be7a1e2a99a\") " pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:19:04.937172 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:04.937103 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-dshm\") pod \"premium-w0-ee876a-kserve-67b45d4948-n6tcp\" (UID: \"8b3ccd4d-549f-4621-908f-5be7a1e2a99a\") " pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:19:04.937244 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:04.937226 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-tmp-dir\") pod \"premium-w0-ee876a-kserve-67b45d4948-n6tcp\" (UID: \"8b3ccd4d-549f-4621-908f-5be7a1e2a99a\") " pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:19:04.937297 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:04.937271 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-model-cache\") pod \"premium-w0-ee876a-kserve-67b45d4948-n6tcp\" (UID: \"8b3ccd4d-549f-4621-908f-5be7a1e2a99a\") " pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:19:05.038350 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:05.038272 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-tmp-dir\") pod \"premium-w0-ee876a-kserve-67b45d4948-n6tcp\" (UID: \"8b3ccd4d-549f-4621-908f-5be7a1e2a99a\") " pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:19:05.038350 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:05.038333 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-model-cache\") pod \"premium-w0-ee876a-kserve-67b45d4948-n6tcp\" (UID: \"8b3ccd4d-549f-4621-908f-5be7a1e2a99a\") " pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:19:05.038524 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:05.038363 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xf5m\" (UniqueName: \"kubernetes.io/projected/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-kube-api-access-9xf5m\") pod \"premium-w0-ee876a-kserve-67b45d4948-n6tcp\" (UID: \"8b3ccd4d-549f-4621-908f-5be7a1e2a99a\") " pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:19:05.038524 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:05.038457 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-home\") pod \"premium-w0-ee876a-kserve-67b45d4948-n6tcp\" (UID: \"8b3ccd4d-549f-4621-908f-5be7a1e2a99a\") " pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:19:05.038524 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:05.038505 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-tls-certs\") pod \"premium-w0-ee876a-kserve-67b45d4948-n6tcp\" (UID: \"8b3ccd4d-549f-4621-908f-5be7a1e2a99a\") " pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:19:05.038678 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:05.038544 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-dshm\") pod \"premium-w0-ee876a-kserve-67b45d4948-n6tcp\" (UID: \"8b3ccd4d-549f-4621-908f-5be7a1e2a99a\") " pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:19:05.038678 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:05.038649 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-tmp-dir\") pod \"premium-w0-ee876a-kserve-67b45d4948-n6tcp\" (UID: \"8b3ccd4d-549f-4621-908f-5be7a1e2a99a\") " pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:19:05.038797 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:05.038708 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-model-cache\") pod \"premium-w0-ee876a-kserve-67b45d4948-n6tcp\" (UID: \"8b3ccd4d-549f-4621-908f-5be7a1e2a99a\") " pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:19:05.038797 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:05.038781 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-home\") pod \"premium-w0-ee876a-kserve-67b45d4948-n6tcp\" (UID: \"8b3ccd4d-549f-4621-908f-5be7a1e2a99a\") " pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:19:05.040841 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:05.040820 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-dshm\") pod \"premium-w0-ee876a-kserve-67b45d4948-n6tcp\" (UID: \"8b3ccd4d-549f-4621-908f-5be7a1e2a99a\") " pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:19:05.041213 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:05.041195 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-tls-certs\") pod \"premium-w0-ee876a-kserve-67b45d4948-n6tcp\" (UID: \"8b3ccd4d-549f-4621-908f-5be7a1e2a99a\") " pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:19:05.046360 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:05.046338 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xf5m\" (UniqueName: \"kubernetes.io/projected/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-kube-api-access-9xf5m\") pod \"premium-w0-ee876a-kserve-67b45d4948-n6tcp\" (UID: \"8b3ccd4d-549f-4621-908f-5be7a1e2a99a\") " pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:19:05.097325 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:05.097301 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:19:05.385494 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:05.385465 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp"] Nov 04 01:19:05.393975 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:19:05.387766 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b3ccd4d_549f_4621_908f_5be7a1e2a99a.slice/crio-7a49bcdc0eb41aa449b2559b8f6266bea7504981100fab0201a54ceb6651596c WatchSource:0}: Error finding container 7a49bcdc0eb41aa449b2559b8f6266bea7504981100fab0201a54ceb6651596c: Status 404 returned error can't find the container with id 7a49bcdc0eb41aa449b2559b8f6266bea7504981100fab0201a54ceb6651596c Nov 04 01:19:05.823706 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:05.823667 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" event={"ID":"8b3ccd4d-549f-4621-908f-5be7a1e2a99a","Type":"ContainerStarted","Data":"05600c56a88be7feb214cc76570f270fba59e1ef37abdd77c664a3c20a413817"} Nov 04 01:19:05.824135 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:05.823718 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" event={"ID":"8b3ccd4d-549f-4621-908f-5be7a1e2a99a","Type":"ContainerStarted","Data":"7a49bcdc0eb41aa449b2559b8f6266bea7504981100fab0201a54ceb6651596c"} Nov 04 01:19:05.848701 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:05.848656 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podStartSLOduration=1.848641626 podStartE2EDuration="1.848641626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:05.846141954 +0000 UTC m=+1230.787990858" watchObservedRunningTime="2026-11-04 01:19:05.848641626 +0000 UTC m=+1230.790490528" Nov 04 01:19:06.098265 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:06.098198 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:19:06.480662 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:19:06.480623 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:19:12.521024 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:12.520990 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7"] Nov 04 01:19:12.527128 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:12.527090 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:19:12.537307 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:12.536889 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7"] Nov 04 01:19:12.620426 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:12.620387 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:19:12.620580 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:12.620546 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:19:12.620644 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:12.620594 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwlfd\" (UniqueName: \"kubernetes.io/projected/7e59c4bd-a00f-41e1-8d1c-610b9d791374-kube-api-access-fwlfd\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:19:12.620687 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:12.620662 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:19:12.620771 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:12.620750 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:19:12.620824 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:12.620780 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:19:12.620877 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:12.620834 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:19:12.620947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:12.620895 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:19:12.621013 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:12.620962 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:19:12.628554 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:12.628534 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-d7e45d-openshift-default-dockercfg-rl5m5\"" Nov 04 01:19:12.721600 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:12.721577 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:19:12.721713 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:12.721615 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwlfd\" (UniqueName: \"kubernetes.io/projected/7e59c4bd-a00f-41e1-8d1c-610b9d791374-kube-api-access-fwlfd\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:19:12.721768 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:12.721746 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:19:12.721831 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:12.721819 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:19:12.721871 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:12.721839 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:19:12.721913 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:12.721865 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:19:12.721978 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:12.721912 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:19:12.721978 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:12.721956 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:19:12.722087 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:12.721983 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:19:12.722249 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:12.722230 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:19:12.722300 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:12.722257 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:19:12.722364 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:12.722323 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:19:12.722420 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:12.722378 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:19:12.722797 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:12.722777 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:19:12.724294 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:12.724269 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:19:12.725530 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:12.725505 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:19:12.729734 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:12.729713 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:19:12.730004 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:12.729986 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwlfd\" (UniqueName: \"kubernetes.io/projected/7e59c4bd-a00f-41e1-8d1c-610b9d791374-kube-api-access-fwlfd\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:19:12.846112 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:12.846058 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:19:13.134355 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:13.134327 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7"] Nov 04 01:19:13.135764 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:19:13.135734 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e59c4bd_a00f_41e1_8d1c_610b9d791374.slice/crio-5aeccf0750bc51d1b19da0fc77d9464e1e7e30403f0dbd5b841e884f07a68a07 WatchSource:0}: Error finding container 5aeccf0750bc51d1b19da0fc77d9464e1e7e30403f0dbd5b841e884f07a68a07: Status 404 returned error can't find the container with id 5aeccf0750bc51d1b19da0fc77d9464e1e7e30403f0dbd5b841e884f07a68a07 Nov 04 01:19:13.138278 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:13.138240 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:13.138375 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:13.138312 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:13.138375 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:13.138348 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:13.879378 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:13.879339 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" event={"ID":"7e59c4bd-a00f-41e1-8d1c-610b9d791374","Type":"ContainerStarted","Data":"7f68e8df09a677e526e8ba13b3b5bd8a1e9949fa96226e762581834255b5d0ff"} Nov 04 01:19:13.879808 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:13.879400 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" event={"ID":"7e59c4bd-a00f-41e1-8d1c-610b9d791374","Type":"ContainerStarted","Data":"5aeccf0750bc51d1b19da0fc77d9464e1e7e30403f0dbd5b841e884f07a68a07"} Nov 04 01:19:13.900590 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:13.900539 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" podStartSLOduration=1.900526307 podStartE2EDuration="1.900526307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:13.899884628 +0000 UTC m=+1238.841733531" watchObservedRunningTime="2026-11-04 01:19:13.900526307 +0000 UTC m=+1238.842375209" Nov 04 01:19:14.851946 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:14.850851 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:19:14.851946 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:14.851381 7059 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:19:14.851946 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:14.851430 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" podUID="7e59c4bd-a00f-41e1-8d1c-610b9d791374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 04 01:19:15.097685 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:15.097653 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:19:15.110914 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:15.110845 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:19:15.857998 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:15.857957 7059 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:19:15.858332 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:15.858303 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" podUID="7e59c4bd-a00f-41e1-8d1c-610b9d791374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 04 01:19:15.904691 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:15.904661 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:19:16.856539 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:16.856486 7059 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:19:16.857152 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:16.857122 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" podUID="7e59c4bd-a00f-41e1-8d1c-610b9d791374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 04 01:19:17.852950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:17.848873 7059 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:19:17.852950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:17.848964 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" podUID="7e59c4bd-a00f-41e1-8d1c-610b9d791374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 04 01:19:18.850426 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:18.850352 7059 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 04 01:19:18.851355 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:18.851322 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" podUID="7e59c4bd-a00f-41e1-8d1c-610b9d791374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 04 01:19:19.908725 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:19.908693 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:19:19.909236 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:19.909115 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:19:19.910020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:19.909987 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:19:36.484169 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:19:36.484115 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:19:41.903466 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:41.903431 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l"] Nov 04 01:19:41.909137 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:41.909115 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:19:41.918955 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:41.918910 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-9c8bf5-openshift-default-dockercfg-4ks7x\"" Nov 04 01:19:41.926994 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:41.926966 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l"] Nov 04 01:19:42.078459 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.078425 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-workload-certs\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:19:42.078459 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.078457 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f8924f-e452-4b2c-8b41-f970764ff963-istio-podinfo\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:19:42.078654 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.078578 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f8924f-e452-4b2c-8b41-f970764ff963-istiod-ca-cert\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:19:42.078654 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.078607 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-credential-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:19:42.078654 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.078625 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-istio-data\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:19:42.078794 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.078774 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-istio-envoy\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:19:42.078830 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.078814 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-workload-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:19:42.078863 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.078834 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0f8924f-e452-4b2c-8b41-f970764ff963-istio-token\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:19:42.079001 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.078981 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6hcp\" (UniqueName: \"kubernetes.io/projected/f0f8924f-e452-4b2c-8b41-f970764ff963-kube-api-access-c6hcp\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:19:42.180400 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.180370 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6hcp\" (UniqueName: \"kubernetes.io/projected/f0f8924f-e452-4b2c-8b41-f970764ff963-kube-api-access-c6hcp\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:19:42.180562 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.180414 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-workload-certs\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:19:42.180562 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.180431 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f8924f-e452-4b2c-8b41-f970764ff963-istio-podinfo\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:19:42.180562 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.180497 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f8924f-e452-4b2c-8b41-f970764ff963-istiod-ca-cert\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:19:42.180562 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.180524 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-credential-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:19:42.180776 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.180661 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-istio-data\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:19:42.180831 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.180791 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-istio-envoy\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:19:42.180888 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.180820 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-workload-certs\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:19:42.180888 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.180841 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-workload-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:19:42.180888 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.180843 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-credential-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:19:42.181079 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.180902 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0f8924f-e452-4b2c-8b41-f970764ff963-istio-token\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:19:42.181130 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.181085 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-istio-data\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:19:42.181197 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.181146 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-workload-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:19:42.181313 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.181288 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f8924f-e452-4b2c-8b41-f970764ff963-istiod-ca-cert\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:19:42.183041 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.183019 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-istio-envoy\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:19:42.183214 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.183194 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f8924f-e452-4b2c-8b41-f970764ff963-istio-podinfo\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:19:42.189343 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.189316 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0f8924f-e452-4b2c-8b41-f970764ff963-istio-token\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:19:42.189466 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.189422 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6hcp\" (UniqueName: \"kubernetes.io/projected/f0f8924f-e452-4b2c-8b41-f970764ff963-kube-api-access-c6hcp\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:19:42.227333 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.227301 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:19:42.284743 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.284707 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw"] Nov 04 01:19:42.292395 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.292368 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:19:42.302949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.302869 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw"] Nov 04 01:19:42.370782 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.370733 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-499ddd-openshift-default-dockercfg-fb8nq\"" Nov 04 01:19:42.371414 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.371394 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876"] Nov 04 01:19:42.375554 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.375535 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:19:42.390004 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.389979 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876"] Nov 04 01:19:42.470950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.470853 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-ca55c1-openshift-default-dockercfg-t5hrc\"" Nov 04 01:19:42.486852 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.486811 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:19:42.486852 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.486863 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49cff42c-6b14-409e-b5a1-9eb70c788da9-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:19:42.487114 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.486896 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:19:42.487114 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.486937 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b925870-1345-4a77-8f3b-2175eb9bb93a-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:19:42.487114 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.487041 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:19:42.487114 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.487065 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:19:42.487114 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.487101 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh92h\" (UniqueName: \"kubernetes.io/projected/5b925870-1345-4a77-8f3b-2175eb9bb93a-kube-api-access-sh92h\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:19:42.487392 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.487160 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49cff42c-6b14-409e-b5a1-9eb70c788da9-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:19:42.487392 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.487190 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:19:42.487392 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.487239 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:19:42.487392 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.487259 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:19:42.487392 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.487362 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:19:42.487661 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.487415 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49cff42c-6b14-409e-b5a1-9eb70c788da9-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:19:42.487661 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.487431 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grfhn\" (UniqueName: \"kubernetes.io/projected/49cff42c-6b14-409e-b5a1-9eb70c788da9-kube-api-access-grfhn\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:19:42.487661 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.487585 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:19:42.487661 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.487627 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:19:42.487867 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.487691 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b925870-1345-4a77-8f3b-2175eb9bb93a-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:19:42.487867 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.487714 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b925870-1345-4a77-8f3b-2175eb9bb93a-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:19:42.537422 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.537393 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l"] Nov 04 01:19:42.538584 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:19:42.538556 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f8924f_e452_4b2c_8b41_f970764ff963.slice/crio-25c24397771825b4d0c117b6f1ce67f36bf4e80b480c6bed9dd39f7ad4b5c74e WatchSource:0}: Error finding container 25c24397771825b4d0c117b6f1ce67f36bf4e80b480c6bed9dd39f7ad4b5c74e: Status 404 returned error can't find the container with id 25c24397771825b4d0c117b6f1ce67f36bf4e80b480c6bed9dd39f7ad4b5c74e Nov 04 01:19:42.540936 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.540887 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:42.541022 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.540966 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:42.541022 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.540998 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:42.589169 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.589142 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:19:42.589285 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.589232 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49cff42c-6b14-409e-b5a1-9eb70c788da9-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:19:42.589352 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.589291 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grfhn\" (UniqueName: \"kubernetes.io/projected/49cff42c-6b14-409e-b5a1-9eb70c788da9-kube-api-access-grfhn\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:19:42.589411 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.589359 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:19:42.589411 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.589387 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:19:42.589522 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.589444 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b925870-1345-4a77-8f3b-2175eb9bb93a-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:19:42.589522 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.589465 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b925870-1345-4a77-8f3b-2175eb9bb93a-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:19:42.589758 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.589522 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:19:42.589758 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.589554 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49cff42c-6b14-409e-b5a1-9eb70c788da9-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:19:42.589758 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.589585 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:19:42.589758 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.589609 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b925870-1345-4a77-8f3b-2175eb9bb93a-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:19:42.589758 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.589664 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:19:42.589758 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.589692 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:19:42.589758 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.589712 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh92h\" (UniqueName: \"kubernetes.io/projected/5b925870-1345-4a77-8f3b-2175eb9bb93a-kube-api-access-sh92h\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:19:42.589758 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.589755 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49cff42c-6b14-409e-b5a1-9eb70c788da9-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:19:42.590105 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.589801 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:19:42.590105 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.589828 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:19:42.590105 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.589849 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:19:42.590105 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.589995 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:19:42.590105 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.590005 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:19:42.590516 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.590149 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49cff42c-6b14-409e-b5a1-9eb70c788da9-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:19:42.590516 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.590223 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:19:42.590516 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.590416 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:19:42.590516 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.590507 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:19:42.590744 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.590512 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:19:42.590804 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.590773 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:19:42.591828 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.591268 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b925870-1345-4a77-8f3b-2175eb9bb93a-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:19:42.591828 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.591321 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:19:42.592600 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.592558 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b925870-1345-4a77-8f3b-2175eb9bb93a-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:19:42.592949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.592900 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:19:42.593189 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.593170 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:19:42.593353 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.593334 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49cff42c-6b14-409e-b5a1-9eb70c788da9-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:19:42.602371 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.602350 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b925870-1345-4a77-8f3b-2175eb9bb93a-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:19:42.602616 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.602593 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49cff42c-6b14-409e-b5a1-9eb70c788da9-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:19:42.603037 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.603013 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh92h\" (UniqueName: \"kubernetes.io/projected/5b925870-1345-4a77-8f3b-2175eb9bb93a-kube-api-access-sh92h\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:19:42.605110 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.605091 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grfhn\" (UniqueName: \"kubernetes.io/projected/49cff42c-6b14-409e-b5a1-9eb70c788da9-kube-api-access-grfhn\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:19:42.620114 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.620092 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:19:42.692734 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.692346 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:19:42.911993 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:19:42.911959 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b925870_1345_4a77_8f3b_2175eb9bb93a.slice/crio-39cb9c650b30eb1deda9be28f0a64b1c0712837dcff7c8ce962f700b3088827b WatchSource:0}: Error finding container 39cb9c650b30eb1deda9be28f0a64b1c0712837dcff7c8ce962f700b3088827b: Status 404 returned error can't find the container with id 39cb9c650b30eb1deda9be28f0a64b1c0712837dcff7c8ce962f700b3088827b Nov 04 01:19:42.912660 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.912639 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw"] Nov 04 01:19:42.914453 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.914414 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:42.914570 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.914484 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:42.914570 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:42.914526 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:43.015359 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:43.015325 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876"] Nov 04 01:19:43.016579 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:19:43.016543 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49cff42c_6b14_409e_b5a1_9eb70c788da9.slice/crio-d667ab40fd561489cd4d4520835597e4138d466ef97bd51a2d5ff1cc9a31af24 WatchSource:0}: Error finding container d667ab40fd561489cd4d4520835597e4138d466ef97bd51a2d5ff1cc9a31af24: Status 404 returned error can't find the container with id d667ab40fd561489cd4d4520835597e4138d466ef97bd51a2d5ff1cc9a31af24 Nov 04 01:19:43.019803 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:43.019759 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:43.019951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:43.019848 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:43.020012 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:43.019959 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:43.100705 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:43.100672 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" event={"ID":"f0f8924f-e452-4b2c-8b41-f970764ff963","Type":"ContainerStarted","Data":"08dfdcd839590cf2ee0e3d7c29edd4feb32fee076c7a6ae91b7745d570b8a82e"} Nov 04 01:19:43.100864 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:43.100723 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" event={"ID":"f0f8924f-e452-4b2c-8b41-f970764ff963","Type":"ContainerStarted","Data":"25c24397771825b4d0c117b6f1ce67f36bf4e80b480c6bed9dd39f7ad4b5c74e"} Nov 04 01:19:43.102153 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:43.102129 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" event={"ID":"49cff42c-6b14-409e-b5a1-9eb70c788da9","Type":"ContainerStarted","Data":"d667ab40fd561489cd4d4520835597e4138d466ef97bd51a2d5ff1cc9a31af24"} Nov 04 01:19:43.104469 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:43.103880 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" event={"ID":"5b925870-1345-4a77-8f3b-2175eb9bb93a","Type":"ContainerStarted","Data":"8da6d1bb6c9da5ab033ee837b326722c8fb4bca4bf7b5c0bf79cc1cce8fa7bda"} Nov 04 01:19:43.104469 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:43.103945 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" event={"ID":"5b925870-1345-4a77-8f3b-2175eb9bb93a","Type":"ContainerStarted","Data":"39cb9c650b30eb1deda9be28f0a64b1c0712837dcff7c8ce962f700b3088827b"} Nov 04 01:19:43.124766 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:43.124725 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" podStartSLOduration=2.124710943 podStartE2EDuration="2.124710943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:43.122514604 +0000 UTC m=+1268.064363519" watchObservedRunningTime="2026-11-04 01:19:43.124710943 +0000 UTC m=+1268.066559845" Nov 04 01:19:43.149949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:43.146849 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" podStartSLOduration=1.146830376 podStartE2EDuration="1.146830376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:43.145432299 +0000 UTC m=+1268.087281215" watchObservedRunningTime="2026-11-04 01:19:43.146830376 +0000 UTC m=+1268.088679280" Nov 04 01:19:43.228213 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:43.228166 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:19:43.229700 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:43.229674 7059 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 04 01:19:43.229820 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:43.229723 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" podUID="f0f8924f-e452-4b2c-8b41-f970764ff963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 04 01:19:43.620849 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:43.620815 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:19:43.622313 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:43.622286 7059 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 04 01:19:43.622460 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:43.622333 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" podUID="5b925870-1345-4a77-8f3b-2175eb9bb93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 04 01:19:44.124845 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:44.124810 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" event={"ID":"49cff42c-6b14-409e-b5a1-9eb70c788da9","Type":"ContainerStarted","Data":"bb8bfcd43e9b13d72b47ab8700ac14fc4d34a7712593d4f2cb2666457f8d177b"} Nov 04 01:19:44.228197 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:44.228172 7059 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 04 01:19:44.228421 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:44.228396 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" podUID="f0f8924f-e452-4b2c-8b41-f970764ff963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 04 01:19:44.621572 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:44.621528 7059 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 04 01:19:44.621822 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:44.621610 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" podUID="5b925870-1345-4a77-8f3b-2175eb9bb93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 04 01:19:44.694629 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:44.694525 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:19:44.697356 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:44.697323 7059 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 04 01:19:44.697573 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:44.697543 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" podUID="49cff42c-6b14-409e-b5a1-9eb70c788da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 04 01:19:45.227872 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:45.227835 7059 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 04 01:19:45.228303 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:45.227891 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" podUID="f0f8924f-e452-4b2c-8b41-f970764ff963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 04 01:19:45.620498 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:45.620404 7059 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 04 01:19:45.620498 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:45.620455 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" podUID="5b925870-1345-4a77-8f3b-2175eb9bb93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 04 01:19:45.699947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:45.699492 7059 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 04 01:19:45.699947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:45.699551 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" podUID="49cff42c-6b14-409e-b5a1-9eb70c788da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 04 01:19:46.227787 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:46.227750 7059 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 04 01:19:46.227998 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:46.227801 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" podUID="f0f8924f-e452-4b2c-8b41-f970764ff963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 04 01:19:46.620888 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:46.620797 7059 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 04 01:19:46.620888 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:46.620867 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" podUID="5b925870-1345-4a77-8f3b-2175eb9bb93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 04 01:19:46.704949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:46.702271 7059 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 04 01:19:46.704949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:46.702330 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" podUID="49cff42c-6b14-409e-b5a1-9eb70c788da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 04 01:19:47.228635 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:47.228411 7059 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 04 01:19:47.228635 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:47.228504 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" podUID="f0f8924f-e452-4b2c-8b41-f970764ff963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 04 01:19:47.621826 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:47.621292 7059 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 04 01:19:47.621826 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:47.621357 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" podUID="5b925870-1345-4a77-8f3b-2175eb9bb93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 04 01:19:48.226948 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:48.226538 7059 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 04 01:19:48.226948 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:48.226639 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" podUID="49cff42c-6b14-409e-b5a1-9eb70c788da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 04 01:19:48.236999 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:48.236312 7059 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 04 01:19:48.320326 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:48.320276 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" podUID="f0f8924f-e452-4b2c-8b41-f970764ff963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 04 01:19:48.621546 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:48.621458 7059 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 04 01:19:48.621685 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:48.621536 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" podUID="5b925870-1345-4a77-8f3b-2175eb9bb93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 04 01:19:48.697482 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:48.697448 7059 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 04 01:19:48.697836 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:48.697795 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" podUID="49cff42c-6b14-409e-b5a1-9eb70c788da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 04 01:19:49.228208 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:49.228171 7059 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 04 01:19:49.228356 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:49.228241 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" podUID="f0f8924f-e452-4b2c-8b41-f970764ff963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 04 01:19:49.621494 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:49.621395 7059 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 04 01:19:49.621494 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:49.621459 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" podUID="5b925870-1345-4a77-8f3b-2175eb9bb93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 04 01:19:49.693097 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:49.693062 7059 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 04 01:19:49.693406 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:49.693119 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" podUID="49cff42c-6b14-409e-b5a1-9eb70c788da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 04 01:19:50.233681 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:50.233644 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:19:50.234065 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:50.234039 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:19:50.234812 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:50.234791 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:19:50.261653 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:50.261603 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" podStartSLOduration=8.261590007 podStartE2EDuration="8.261590007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:44.160275901 +0000 UTC m=+1269.102124817" watchObservedRunningTime="2026-11-04 01:19:50.261590007 +0000 UTC m=+1275.203438912" Nov 04 01:19:50.625794 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:50.625718 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:19:50.626285 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:50.626055 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:19:50.626878 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:50.626838 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:19:50.699000 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:50.698973 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:19:50.699803 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:50.699782 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:19:50.700557 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:50.700539 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:19:57.184900 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.184865 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2"] Nov 04 01:19:57.189051 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.189033 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:19:57.197963 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.197578 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-099dd9-openshift-default-dockercfg-ntzzj\"" Nov 04 01:19:57.204762 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.204736 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2"] Nov 04 01:19:57.205319 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.205299 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-credential-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:19:57.205396 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.205342 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bwrs\" (UniqueName: \"kubernetes.io/projected/b59977fd-6792-40e9-9070-1f04beb3f0bf-kube-api-access-4bwrs\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:19:57.205455 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.205432 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-workload-certs\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:19:57.205491 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.205472 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b59977fd-6792-40e9-9070-1f04beb3f0bf-istio-token\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:19:57.205621 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.205599 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b59977fd-6792-40e9-9070-1f04beb3f0bf-istio-podinfo\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:19:57.205689 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.205639 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-workload-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:19:57.205689 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.205657 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-istio-data\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:19:57.205792 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.205698 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b59977fd-6792-40e9-9070-1f04beb3f0bf-istiod-ca-cert\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:19:57.205892 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.205874 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-istio-envoy\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:19:57.307004 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.306969 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-workload-certs\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:19:57.307004 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.307010 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b59977fd-6792-40e9-9070-1f04beb3f0bf-istio-token\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:19:57.307235 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.307064 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b59977fd-6792-40e9-9070-1f04beb3f0bf-istio-podinfo\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:19:57.307235 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.307089 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-workload-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:19:57.307235 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.307111 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-istio-data\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:19:57.307235 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.307135 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b59977fd-6792-40e9-9070-1f04beb3f0bf-istiod-ca-cert\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:19:57.307235 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.307194 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-istio-envoy\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:19:57.307480 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.307297 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-credential-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:19:57.307480 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.307450 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-workload-certs\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:19:57.307901 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.307548 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bwrs\" (UniqueName: \"kubernetes.io/projected/b59977fd-6792-40e9-9070-1f04beb3f0bf-kube-api-access-4bwrs\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:19:57.308066 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.307552 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-workload-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:19:57.308066 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.307953 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b59977fd-6792-40e9-9070-1f04beb3f0bf-istiod-ca-cert\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:19:57.308066 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.307778 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-credential-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:19:57.308066 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.308043 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-istio-data\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:19:57.310024 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.309994 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b59977fd-6792-40e9-9070-1f04beb3f0bf-istio-podinfo\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:19:57.310266 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.310244 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-istio-envoy\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:19:57.323584 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.323559 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b59977fd-6792-40e9-9070-1f04beb3f0bf-istio-token\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:19:57.324818 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.324793 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bwrs\" (UniqueName: \"kubernetes.io/projected/b59977fd-6792-40e9-9070-1f04beb3f0bf-kube-api-access-4bwrs\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:19:57.435557 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.435473 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp"] Nov 04 01:19:57.441013 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.440991 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:19:57.451719 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.451696 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ee876a-openshift-default-dockercfg-v2pll\"" Nov 04 01:19:57.455617 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.455593 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp"] Nov 04 01:19:57.506132 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.506108 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:19:57.509482 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.509205 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-workload-certs\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:19:57.509482 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.509252 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-workload-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:19:57.509482 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.509368 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-credential-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:19:57.509482 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.509398 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istio-token\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:19:57.509482 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.509460 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istiod-ca-cert\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:19:57.509776 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.509502 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istio-data\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:19:57.509776 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.509525 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twbdx\" (UniqueName: \"kubernetes.io/projected/8d6194a6-04a4-4d8d-8017-de919dc36fa5-kube-api-access-twbdx\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:19:57.509776 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.509576 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istio-envoy\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:19:57.509776 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.509617 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istio-podinfo\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:19:57.610776 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.610740 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-workload-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:19:57.610962 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.610825 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-credential-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:19:57.610962 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.610849 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istio-token\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:19:57.610962 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.610883 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istiod-ca-cert\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:19:57.610962 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.610915 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istio-data\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:19:57.610962 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.610962 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twbdx\" (UniqueName: \"kubernetes.io/projected/8d6194a6-04a4-4d8d-8017-de919dc36fa5-kube-api-access-twbdx\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:19:57.611271 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.611007 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istio-envoy\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:19:57.611271 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.611061 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istio-podinfo\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:19:57.611271 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.611085 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-workload-certs\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:19:57.611271 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.611202 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-workload-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:19:57.611484 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.611346 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-workload-certs\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:19:57.611541 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.611521 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istio-data\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:19:57.611616 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.611599 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istiod-ca-cert\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:19:57.612072 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.612050 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-credential-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:19:57.614320 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.614298 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istio-podinfo\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:19:57.615199 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.615141 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istio-envoy\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:19:57.620277 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.620248 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istio-token\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:19:57.620837 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.620814 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twbdx\" (UniqueName: \"kubernetes.io/projected/8d6194a6-04a4-4d8d-8017-de919dc36fa5-kube-api-access-twbdx\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:19:57.760302 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.760217 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:19:57.793598 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.793569 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2"] Nov 04 01:19:57.794133 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:19:57.794107 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb59977fd_6792_40e9_9070_1f04beb3f0bf.slice/crio-0371cf25dee10ab30827bf7f9ee0e23280cd20f51b93c4dcd0b8e1df6cdfc099 WatchSource:0}: Error finding container 0371cf25dee10ab30827bf7f9ee0e23280cd20f51b93c4dcd0b8e1df6cdfc099: Status 404 returned error can't find the container with id 0371cf25dee10ab30827bf7f9ee0e23280cd20f51b93c4dcd0b8e1df6cdfc099 Nov 04 01:19:57.796615 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.796585 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:57.796727 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.796649 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:57.796727 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:57.796700 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:58.054625 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:58.054600 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp"] Nov 04 01:19:58.055671 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:19:58.055640 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d6194a6_04a4_4d8d_8017_de919dc36fa5.slice/crio-c07916e78a9542955eb54eafde118f4533966521a9ba1285c465e262e5a6905d WatchSource:0}: Error finding container c07916e78a9542955eb54eafde118f4533966521a9ba1285c465e262e5a6905d: Status 404 returned error can't find the container with id c07916e78a9542955eb54eafde118f4533966521a9ba1285c465e262e5a6905d Nov 04 01:19:58.058948 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:58.058895 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:58.059042 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:58.058976 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:58.059042 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:58.059007 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:19:58.407391 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:58.407300 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" event={"ID":"b59977fd-6792-40e9-9070-1f04beb3f0bf","Type":"ContainerStarted","Data":"57d4c0a099c9e69dfe379431eebf2798321cbea14f78e4720e56dc3fbcf7ac91"} Nov 04 01:19:58.407391 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:58.407362 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" event={"ID":"b59977fd-6792-40e9-9070-1f04beb3f0bf","Type":"ContainerStarted","Data":"0371cf25dee10ab30827bf7f9ee0e23280cd20f51b93c4dcd0b8e1df6cdfc099"} Nov 04 01:19:58.409309 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:58.409284 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" event={"ID":"8d6194a6-04a4-4d8d-8017-de919dc36fa5","Type":"ContainerStarted","Data":"89d5d9573cb883da8e2894ebc53363f63ee8d469a14dbfbdc8e7a1c3ac69ab71"} Nov 04 01:19:58.409500 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:58.409473 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" event={"ID":"8d6194a6-04a4-4d8d-8017-de919dc36fa5","Type":"ContainerStarted","Data":"c07916e78a9542955eb54eafde118f4533966521a9ba1285c465e262e5a6905d"} Nov 04 01:19:58.434902 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:58.432096 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" podStartSLOduration=1.432076999 podStartE2EDuration="1.432076999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:58.431057319 +0000 UTC m=+1283.372906224" watchObservedRunningTime="2026-11-04 01:19:58.432076999 +0000 UTC m=+1283.373925905" Nov 04 01:19:58.483546 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:58.483486 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" podStartSLOduration=1.483467866 podStartE2EDuration="1.483467866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:58.48204675 +0000 UTC m=+1283.423895653" watchObservedRunningTime="2026-11-04 01:19:58.483467866 +0000 UTC m=+1283.425316770" Nov 04 01:19:58.506797 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:58.506767 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:19:58.508231 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:58.508200 7059 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 04 01:19:58.508350 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:58.508258 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" podUID="b59977fd-6792-40e9-9070-1f04beb3f0bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 04 01:19:58.771122 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:58.771064 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:19:59.506795 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:59.506748 7059 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 04 01:19:59.507282 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:59.506831 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" podUID="b59977fd-6792-40e9-9070-1f04beb3f0bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 04 01:19:59.792146 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:59.792054 7059 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 04 01:19:59.792146 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:19:59.792131 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" podUID="8d6194a6-04a4-4d8d-8017-de919dc36fa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 04 01:20:00.507576 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:00.507536 7059 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 04 01:20:00.507999 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:00.507600 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" podUID="b59977fd-6792-40e9-9070-1f04beb3f0bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 04 01:20:00.795030 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:00.770309 7059 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 04 01:20:00.795030 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:00.770388 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" podUID="8d6194a6-04a4-4d8d-8017-de919dc36fa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 04 01:20:01.507379 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:01.507346 7059 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 04 01:20:01.507599 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:01.507578 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" podUID="b59977fd-6792-40e9-9070-1f04beb3f0bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 04 01:20:01.763001 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:01.762273 7059 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 04 01:20:01.763001 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:01.762345 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" podUID="8d6194a6-04a4-4d8d-8017-de919dc36fa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 04 01:20:02.507187 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:02.507158 7059 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 04 01:20:02.507415 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:02.507395 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" podUID="b59977fd-6792-40e9-9070-1f04beb3f0bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 04 01:20:02.762000 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:02.761327 7059 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 04 01:20:02.762000 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:02.761389 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" podUID="8d6194a6-04a4-4d8d-8017-de919dc36fa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 04 01:20:03.510973 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:03.510122 7059 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 04 01:20:03.510973 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:03.510283 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" podUID="b59977fd-6792-40e9-9070-1f04beb3f0bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 04 01:20:03.777028 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:03.776076 7059 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 04 01:20:03.777028 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:03.776136 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" podUID="8d6194a6-04a4-4d8d-8017-de919dc36fa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 04 01:20:04.507292 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:04.507256 7059 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 04 01:20:04.507447 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:04.507316 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" podUID="b59977fd-6792-40e9-9070-1f04beb3f0bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 04 01:20:05.512388 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:05.512359 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:20:05.512803 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:05.512678 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:20:05.513661 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:05.513639 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:20:05.803145 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:05.803053 7059 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:20:05.803145 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:05.803115 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" podUID="8d6194a6-04a4-4d8d-8017-de919dc36fa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:05.887352 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:05.887321 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:20:05.887981 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:05.887739 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:20:05.889208 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:05.889178 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:20:06.480683 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:20:06.480648 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:20:19.231080 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:19.231040 7059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bprgn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:20:19.231590 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:19.231119 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" podUID="5e65c0d9-95d9-43a6-b77f-bd41b3fd6383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:19.264133 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:19.264013 7059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-ddd7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:20:19.264133 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:19.264075 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" podUID="26881d61-360e-43bb-83f8-3122e0026e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:27.002277 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.002244 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c"] Nov 04 01:20:27.006567 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.006548 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:20:27.025286 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.025261 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c"] Nov 04 01:20:27.112847 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.112814 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istio-envoy\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:20:27.113040 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.112852 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istio-data\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:20:27.113040 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.112878 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkqjw\" (UniqueName: \"kubernetes.io/projected/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-kube-api-access-dkqjw\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:20:27.113121 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.113044 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-credential-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:20:27.113121 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.113073 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-workload-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:20:27.113211 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.113185 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-workload-certs\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:20:27.113246 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.113232 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istio-token\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:20:27.113312 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.113295 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istiod-ca-cert\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:20:27.113368 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.113354 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istio-podinfo\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:20:27.215172 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.215142 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istiod-ca-cert\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:20:27.215339 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.215199 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istio-podinfo\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:20:27.215339 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.215295 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istio-envoy\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:20:27.215339 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.215328 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istio-data\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:20:27.215583 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.215373 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkqjw\" (UniqueName: \"kubernetes.io/projected/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-kube-api-access-dkqjw\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:20:27.215583 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.215428 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-credential-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:20:27.215583 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.215456 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-workload-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:20:27.215583 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.215537 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-workload-certs\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:20:27.215853 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.215654 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istio-token\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:20:27.216053 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.216018 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-credential-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:20:27.216319 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.216274 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istio-data\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:20:27.216428 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.216325 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istiod-ca-cert\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:20:27.216521 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.216433 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-workload-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:20:27.216521 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.216481 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-workload-certs\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:20:27.218606 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.218575 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istio-podinfo\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:20:27.218714 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.218610 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istio-envoy\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:20:27.225320 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.225300 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkqjw\" (UniqueName: \"kubernetes.io/projected/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-kube-api-access-dkqjw\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:20:27.225480 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.225305 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istio-token\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:20:27.325276 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.325211 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:20:27.544581 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.544548 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb"] Nov 04 01:20:27.549112 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.549086 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:20:27.560809 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.560781 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb"] Nov 04 01:20:27.562178 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.562162 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-586541-openshift-default-dockercfg-tgz2p\"" Nov 04 01:20:27.616726 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.616698 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c"] Nov 04 01:20:27.617258 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:20:27.617237 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9e034fc_c636_44dd_a58c_b8cf6d3a5734.slice/crio-d4c3e407432dc4ad7309aba478b6f734900d93ea775a2d89de9fc3ec262ba90e WatchSource:0}: Error finding container d4c3e407432dc4ad7309aba478b6f734900d93ea775a2d89de9fc3ec262ba90e: Status 404 returned error can't find the container with id d4c3e407432dc4ad7309aba478b6f734900d93ea775a2d89de9fc3ec262ba90e Nov 04 01:20:27.619757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.619724 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:20:27.619831 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.619789 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:20:27.619831 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.619817 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:20:27.691342 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.691315 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" event={"ID":"f9e034fc-c636-44dd-a58c-b8cf6d3a5734","Type":"ContainerStarted","Data":"d4c3e407432dc4ad7309aba478b6f734900d93ea775a2d89de9fc3ec262ba90e"} Nov 04 01:20:27.725181 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.720241 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-istio-envoy\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:20:27.725181 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.720294 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-workload-certs\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:20:27.725181 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.720404 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2m84\" (UniqueName: \"kubernetes.io/projected/8368cee7-3404-4ae8-b6de-00b3434472d6-kube-api-access-k2m84\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:20:27.725181 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.720468 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8368cee7-3404-4ae8-b6de-00b3434472d6-istio-token\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:20:27.725181 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.720560 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-istio-data\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:20:27.725181 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.720596 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8368cee7-3404-4ae8-b6de-00b3434472d6-istio-podinfo\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:20:27.725181 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.720651 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-credential-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:20:27.725181 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.720735 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-workload-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:20:27.725181 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.720799 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8368cee7-3404-4ae8-b6de-00b3434472d6-istiod-ca-cert\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:20:27.822542 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.822509 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-workload-certs\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:20:27.822749 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.822554 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2m84\" (UniqueName: \"kubernetes.io/projected/8368cee7-3404-4ae8-b6de-00b3434472d6-kube-api-access-k2m84\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:20:27.822749 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.822606 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8368cee7-3404-4ae8-b6de-00b3434472d6-istio-token\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:20:27.822749 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.822653 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-istio-data\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:20:27.822749 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.822694 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8368cee7-3404-4ae8-b6de-00b3434472d6-istio-podinfo\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:20:27.822749 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.822740 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-credential-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:20:27.823064 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.822828 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-workload-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:20:27.823064 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.822892 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8368cee7-3404-4ae8-b6de-00b3434472d6-istiod-ca-cert\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:20:27.823064 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.823003 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-workload-certs\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:20:27.823215 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.823089 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-istio-envoy\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:20:27.823276 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.823255 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-credential-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:20:27.823466 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.823440 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-istio-data\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:20:27.823552 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.823510 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-workload-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:20:27.823720 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.823701 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8368cee7-3404-4ae8-b6de-00b3434472d6-istiod-ca-cert\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:20:27.825461 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.825444 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8368cee7-3404-4ae8-b6de-00b3434472d6-istio-podinfo\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:20:27.825532 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.825492 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-istio-envoy\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:20:27.832953 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.832236 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8368cee7-3404-4ae8-b6de-00b3434472d6-istio-token\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:20:27.832953 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.832369 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2m84\" (UniqueName: \"kubernetes.io/projected/8368cee7-3404-4ae8-b6de-00b3434472d6-kube-api-access-k2m84\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:20:27.866437 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:27.866378 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:20:28.156134 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:28.156104 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb"] Nov 04 01:20:28.156979 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:20:28.156911 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8368cee7_3404_4ae8_b6de_00b3434472d6.slice/crio-f5f8c7a464c49d82e10be5ee78e1b78b662068831b06afebf55521babafc7dd4 WatchSource:0}: Error finding container f5f8c7a464c49d82e10be5ee78e1b78b662068831b06afebf55521babafc7dd4: Status 404 returned error can't find the container with id f5f8c7a464c49d82e10be5ee78e1b78b662068831b06afebf55521babafc7dd4 Nov 04 01:20:28.161355 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:28.161312 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:20:28.161480 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:28.161384 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:20:28.161480 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:28.161411 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:20:28.700830 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:28.700788 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" event={"ID":"8368cee7-3404-4ae8-b6de-00b3434472d6","Type":"ContainerStarted","Data":"1b21a3a4658177e65b799ef632194adf50a8c3e0547850b554001f6aeb89fdd6"} Nov 04 01:20:28.701052 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:28.700894 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" event={"ID":"8368cee7-3404-4ae8-b6de-00b3434472d6","Type":"ContainerStarted","Data":"f5f8c7a464c49d82e10be5ee78e1b78b662068831b06afebf55521babafc7dd4"} Nov 04 01:20:28.702460 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:28.702430 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" event={"ID":"f9e034fc-c636-44dd-a58c-b8cf6d3a5734","Type":"ContainerStarted","Data":"350cc5ac2d6f7c94547d73b5a37f11f40862fa5f593a1504260c1857d142c724"} Nov 04 01:20:28.776347 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:28.776284 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" podStartSLOduration=1.7566322429999999 podStartE2EDuration="1.756632243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:28.735336731 +0000 UTC m=+1313.677185663" watchObservedRunningTime="2026-11-04 01:20:28.756632243 +0000 UTC m=+1313.698481146" Nov 04 01:20:28.783239 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:28.783145 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" podStartSLOduration=2.783125284 podStartE2EDuration="2.783125284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:28.780062702 +0000 UTC m=+1313.721911594" watchObservedRunningTime="2026-11-04 01:20:28.783125284 +0000 UTC m=+1313.724974188" Nov 04 01:20:28.885948 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:28.875017 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:20:28.904006 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:28.903499 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz"] Nov 04 01:20:28.908711 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:28.908644 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:20:28.920307 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:28.920272 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-586541\"/\"distinct-w1-586541-kserve-self-signed-certs\"" Nov 04 01:20:28.923862 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:28.923771 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz"] Nov 04 01:20:29.064011 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:29.063762 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad50a174-edcc-4898-b072-90d6c9246859-tmp-dir\") pod \"distinct-w1-586541-kserve-6bd6fc799-59vbz\" (UID: \"ad50a174-edcc-4898-b072-90d6c9246859\") " pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:20:29.064011 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:29.063817 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad50a174-edcc-4898-b072-90d6c9246859-home\") pod \"distinct-w1-586541-kserve-6bd6fc799-59vbz\" (UID: \"ad50a174-edcc-4898-b072-90d6c9246859\") " pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:20:29.064011 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:29.063911 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6589k\" (UniqueName: \"kubernetes.io/projected/ad50a174-edcc-4898-b072-90d6c9246859-kube-api-access-6589k\") pod \"distinct-w1-586541-kserve-6bd6fc799-59vbz\" (UID: \"ad50a174-edcc-4898-b072-90d6c9246859\") " pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:20:29.064318 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:29.064020 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad50a174-edcc-4898-b072-90d6c9246859-dshm\") pod \"distinct-w1-586541-kserve-6bd6fc799-59vbz\" (UID: \"ad50a174-edcc-4898-b072-90d6c9246859\") " pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:20:29.064318 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:29.064056 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad50a174-edcc-4898-b072-90d6c9246859-tls-certs\") pod \"distinct-w1-586541-kserve-6bd6fc799-59vbz\" (UID: \"ad50a174-edcc-4898-b072-90d6c9246859\") " pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:20:29.064318 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:29.064102 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad50a174-edcc-4898-b072-90d6c9246859-model-cache\") pod \"distinct-w1-586541-kserve-6bd6fc799-59vbz\" (UID: \"ad50a174-edcc-4898-b072-90d6c9246859\") " pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:20:29.173947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:29.173879 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad50a174-edcc-4898-b072-90d6c9246859-model-cache\") pod \"distinct-w1-586541-kserve-6bd6fc799-59vbz\" (UID: \"ad50a174-edcc-4898-b072-90d6c9246859\") " pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:20:29.174437 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:29.174073 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad50a174-edcc-4898-b072-90d6c9246859-tmp-dir\") pod \"distinct-w1-586541-kserve-6bd6fc799-59vbz\" (UID: \"ad50a174-edcc-4898-b072-90d6c9246859\") " pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:20:29.174437 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:29.174106 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad50a174-edcc-4898-b072-90d6c9246859-home\") pod \"distinct-w1-586541-kserve-6bd6fc799-59vbz\" (UID: \"ad50a174-edcc-4898-b072-90d6c9246859\") " pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:20:29.174437 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:29.174198 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6589k\" (UniqueName: \"kubernetes.io/projected/ad50a174-edcc-4898-b072-90d6c9246859-kube-api-access-6589k\") pod \"distinct-w1-586541-kserve-6bd6fc799-59vbz\" (UID: \"ad50a174-edcc-4898-b072-90d6c9246859\") " pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:20:29.174437 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:29.174279 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad50a174-edcc-4898-b072-90d6c9246859-dshm\") pod \"distinct-w1-586541-kserve-6bd6fc799-59vbz\" (UID: \"ad50a174-edcc-4898-b072-90d6c9246859\") " pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:20:29.174437 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:29.174316 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad50a174-edcc-4898-b072-90d6c9246859-tls-certs\") pod \"distinct-w1-586541-kserve-6bd6fc799-59vbz\" (UID: \"ad50a174-edcc-4898-b072-90d6c9246859\") " pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:20:29.181945 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:29.175205 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad50a174-edcc-4898-b072-90d6c9246859-home\") pod \"distinct-w1-586541-kserve-6bd6fc799-59vbz\" (UID: \"ad50a174-edcc-4898-b072-90d6c9246859\") " pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:20:29.181945 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:29.175465 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad50a174-edcc-4898-b072-90d6c9246859-model-cache\") pod \"distinct-w1-586541-kserve-6bd6fc799-59vbz\" (UID: \"ad50a174-edcc-4898-b072-90d6c9246859\") " pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:20:29.181945 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:29.175688 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad50a174-edcc-4898-b072-90d6c9246859-tmp-dir\") pod \"distinct-w1-586541-kserve-6bd6fc799-59vbz\" (UID: \"ad50a174-edcc-4898-b072-90d6c9246859\") " pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:20:29.189947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:29.184510 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad50a174-edcc-4898-b072-90d6c9246859-dshm\") pod \"distinct-w1-586541-kserve-6bd6fc799-59vbz\" (UID: \"ad50a174-edcc-4898-b072-90d6c9246859\") " pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:20:29.206949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:29.206126 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad50a174-edcc-4898-b072-90d6c9246859-tls-certs\") pod \"distinct-w1-586541-kserve-6bd6fc799-59vbz\" (UID: \"ad50a174-edcc-4898-b072-90d6c9246859\") " pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:20:29.207910 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:29.207881 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6589k\" (UniqueName: \"kubernetes.io/projected/ad50a174-edcc-4898-b072-90d6c9246859-kube-api-access-6589k\") pod \"distinct-w1-586541-kserve-6bd6fc799-59vbz\" (UID: \"ad50a174-edcc-4898-b072-90d6c9246859\") " pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:20:29.290254 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:29.290217 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:20:29.326432 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:29.326313 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:20:29.328295 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:29.328137 7059 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 04 01:20:29.328295 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:29.328205 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" podUID="f9e034fc-c636-44dd-a58c-b8cf6d3a5734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 04 01:20:29.993436 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:29.917062 7059 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 04 01:20:29.993436 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:29.917124 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" podUID="8368cee7-3404-4ae8-b6de-00b3434472d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 04 01:20:30.187961 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:30.187256 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz"] Nov 04 01:20:30.248004 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:20:30.247223 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad50a174_edcc_4898_b072_90d6c9246859.slice/crio-ca51bfaf8404d327e5b7814cde58c096dd495e948a2b92db6b0f5233a8e662f9 WatchSource:0}: Error finding container ca51bfaf8404d327e5b7814cde58c096dd495e948a2b92db6b0f5233a8e662f9: Status 404 returned error can't find the container with id ca51bfaf8404d327e5b7814cde58c096dd495e948a2b92db6b0f5233a8e662f9 Nov 04 01:20:30.340058 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:30.340024 7059 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 04 01:20:30.340307 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:30.340278 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" podUID="f9e034fc-c636-44dd-a58c-b8cf6d3a5734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 04 01:20:30.684066 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:30.684021 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:30.847177 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:30.847065 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 04 01:20:30.878956 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:30.876229 7059 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 04 01:20:30.878956 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:30.876285 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" podUID="8368cee7-3404-4ae8-b6de-00b3434472d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 04 01:20:30.909012 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:30.908821 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:31.019951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:31.017761 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" event={"ID":"ad50a174-edcc-4898-b072-90d6c9246859","Type":"ContainerStarted","Data":"ca51bfaf8404d327e5b7814cde58c096dd495e948a2b92db6b0f5233a8e662f9"} Nov 04 01:20:31.364963 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:31.362349 7059 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 04 01:20:31.364963 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:31.362423 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" podUID="f9e034fc-c636-44dd-a58c-b8cf6d3a5734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 04 01:20:31.725575 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:31.725028 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:31.864395 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:31.864340 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 04 01:20:31.865108 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:31.865088 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:20:31.880000 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:31.872842 7059 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 04 01:20:31.880000 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:31.872889 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" podUID="8368cee7-3404-4ae8-b6de-00b3434472d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 04 01:20:31.928373 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:31.928344 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:20:32.028032 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:32.028001 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" event={"ID":"ad50a174-edcc-4898-b072-90d6c9246859","Type":"ContainerStarted","Data":"713e989f10ddfc20e3b2695c179d01563a68536411955099c1df730b0e2b265c"} Nov 04 01:20:32.064951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:32.064250 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podStartSLOduration=4.064229027 podStartE2EDuration="4.064229027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:32.050324492 +0000 UTC m=+1316.992173396" watchObservedRunningTime="2026-11-04 01:20:32.064229027 +0000 UTC m=+1317.006077931" Nov 04 01:20:32.293088 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:32.293050 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:20:32.326085 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:32.326050 7059 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 04 01:20:32.326222 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:32.326108 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" podUID="f9e034fc-c636-44dd-a58c-b8cf6d3a5734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 04 01:20:32.870944 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:32.870230 7059 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 04 01:20:32.870944 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:32.870303 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" podUID="8368cee7-3404-4ae8-b6de-00b3434472d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 04 01:20:33.325840 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:33.325805 7059 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 04 01:20:33.326042 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:33.325871 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" podUID="f9e034fc-c636-44dd-a58c-b8cf6d3a5734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 04 01:20:33.882678 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:33.882628 7059 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 04 01:20:33.883232 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:33.882704 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" podUID="8368cee7-3404-4ae8-b6de-00b3434472d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 04 01:20:34.326113 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:34.326081 7059 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 04 01:20:34.326265 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:34.326136 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" podUID="f9e034fc-c636-44dd-a58c-b8cf6d3a5734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 04 01:20:34.868247 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:34.868210 7059 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 04 01:20:34.868501 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:34.868478 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" podUID="8368cee7-3404-4ae8-b6de-00b3434472d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 04 01:20:35.901300 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:35.901266 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:20:35.905826 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:35.905801 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:20:35.908523 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:35.908501 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:20:36.292195 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:36.292160 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:20:36.292707 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:36.292687 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:20:36.293563 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:36.293540 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:20:36.491005 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:20:36.490969 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:20:39.291751 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:39.291723 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:20:39.305056 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:39.305021 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:20:40.118637 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:40.118609 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:20:42.817754 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:42.817718 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx"] Nov 04 01:20:42.824624 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:42.824602 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:20:42.836408 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:42.836377 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx"] Nov 04 01:20:42.836545 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:42.836418 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-499ddd\"/\"openshift-service-ca.crt\"" Nov 04 01:20:42.836545 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:42.836479 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-499ddd\"/\"kube-root-ca.crt\"" Nov 04 01:20:42.836743 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:42.836542 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-499ddd\"/\"default-dockercfg-vczgh\"" Nov 04 01:20:42.836743 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:42.836556 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-499ddd\"/\"distinct-w6-499ddd-kserve-self-signed-certs\"" Nov 04 01:20:42.967559 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:42.966852 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/001c1106-c03c-4986-8082-cd7f161c8817-home\") pod \"distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx\" (UID: \"001c1106-c03c-4986-8082-cd7f161c8817\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:20:42.967559 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:42.967046 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z65cs\" (UniqueName: \"kubernetes.io/projected/001c1106-c03c-4986-8082-cd7f161c8817-kube-api-access-z65cs\") pod \"distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx\" (UID: \"001c1106-c03c-4986-8082-cd7f161c8817\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:20:42.967559 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:42.967150 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/001c1106-c03c-4986-8082-cd7f161c8817-model-cache\") pod \"distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx\" (UID: \"001c1106-c03c-4986-8082-cd7f161c8817\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:20:42.967559 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:42.967200 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/001c1106-c03c-4986-8082-cd7f161c8817-tmp-dir\") pod \"distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx\" (UID: \"001c1106-c03c-4986-8082-cd7f161c8817\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:20:42.967559 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:42.967264 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/001c1106-c03c-4986-8082-cd7f161c8817-dshm\") pod \"distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx\" (UID: \"001c1106-c03c-4986-8082-cd7f161c8817\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:20:42.967559 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:42.967330 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/001c1106-c03c-4986-8082-cd7f161c8817-tls-certs\") pod \"distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx\" (UID: \"001c1106-c03c-4986-8082-cd7f161c8817\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:20:43.068346 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:43.068259 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z65cs\" (UniqueName: \"kubernetes.io/projected/001c1106-c03c-4986-8082-cd7f161c8817-kube-api-access-z65cs\") pod \"distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx\" (UID: \"001c1106-c03c-4986-8082-cd7f161c8817\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:20:43.068346 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:43.068329 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/001c1106-c03c-4986-8082-cd7f161c8817-model-cache\") pod \"distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx\" (UID: \"001c1106-c03c-4986-8082-cd7f161c8817\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:20:43.068587 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:43.068369 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/001c1106-c03c-4986-8082-cd7f161c8817-tmp-dir\") pod \"distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx\" (UID: \"001c1106-c03c-4986-8082-cd7f161c8817\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:20:43.068587 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:43.068413 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/001c1106-c03c-4986-8082-cd7f161c8817-dshm\") pod \"distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx\" (UID: \"001c1106-c03c-4986-8082-cd7f161c8817\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:20:43.068587 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:43.068538 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/001c1106-c03c-4986-8082-cd7f161c8817-tls-certs\") pod \"distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx\" (UID: \"001c1106-c03c-4986-8082-cd7f161c8817\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:20:43.068727 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:43.068637 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/001c1106-c03c-4986-8082-cd7f161c8817-home\") pod \"distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx\" (UID: \"001c1106-c03c-4986-8082-cd7f161c8817\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:20:43.068880 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:43.068858 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/001c1106-c03c-4986-8082-cd7f161c8817-tmp-dir\") pod \"distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx\" (UID: \"001c1106-c03c-4986-8082-cd7f161c8817\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:20:43.069011 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:43.068989 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/001c1106-c03c-4986-8082-cd7f161c8817-model-cache\") pod \"distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx\" (UID: \"001c1106-c03c-4986-8082-cd7f161c8817\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:20:43.069068 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:43.069025 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/001c1106-c03c-4986-8082-cd7f161c8817-home\") pod \"distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx\" (UID: \"001c1106-c03c-4986-8082-cd7f161c8817\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:20:43.071031 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:43.071008 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/001c1106-c03c-4986-8082-cd7f161c8817-dshm\") pod \"distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx\" (UID: \"001c1106-c03c-4986-8082-cd7f161c8817\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:20:43.071240 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:43.071222 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/001c1106-c03c-4986-8082-cd7f161c8817-tls-certs\") pod \"distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx\" (UID: \"001c1106-c03c-4986-8082-cd7f161c8817\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:20:43.076519 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:43.076499 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z65cs\" (UniqueName: \"kubernetes.io/projected/001c1106-c03c-4986-8082-cd7f161c8817-kube-api-access-z65cs\") pod \"distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx\" (UID: \"001c1106-c03c-4986-8082-cd7f161c8817\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:20:43.139076 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:43.139046 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:20:43.433634 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:43.433612 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx"] Nov 04 01:20:43.434702 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:20:43.434678 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod001c1106_c03c_4986_8082_cd7f161c8817.slice/crio-421019c7148637aaad49e9de94232351d3d5577eed950710628cdddaa7ddde33 WatchSource:0}: Error finding container 421019c7148637aaad49e9de94232351d3d5577eed950710628cdddaa7ddde33: Status 404 returned error can't find the container with id 421019c7148637aaad49e9de94232351d3d5577eed950710628cdddaa7ddde33 Nov 04 01:20:44.136090 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:44.136046 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" event={"ID":"001c1106-c03c-4986-8082-cd7f161c8817","Type":"ContainerStarted","Data":"dce536f9fb0717644efe768bb8c2e85ef44ba5ecaa3d562d8853587d858b8bf4"} Nov 04 01:20:44.136635 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:44.136598 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" event={"ID":"001c1106-c03c-4986-8082-cd7f161c8817","Type":"ContainerStarted","Data":"421019c7148637aaad49e9de94232351d3d5577eed950710628cdddaa7ddde33"} Nov 04 01:20:44.139465 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:44.139438 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:20:44.157870 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:44.157818 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podStartSLOduration=2.157806548 podStartE2EDuration="2.157806548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:44.154848661 +0000 UTC m=+1329.096697568" watchObservedRunningTime="2026-11-04 01:20:44.157806548 +0000 UTC m=+1329.099655450" Nov 04 01:20:53.139308 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:53.139274 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:20:53.151633 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:53.151601 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:20:53.287774 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:53.287749 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:20:57.289998 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.289462 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9"] Nov 04 01:20:57.294751 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.294727 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:20:57.310906 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.310880 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9"] Nov 04 01:20:57.451797 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.451759 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-credential-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:20:57.451986 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.451817 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istio-token\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:20:57.451986 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.451881 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istio-data\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:20:57.452075 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.451977 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istio-envoy\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:20:57.452075 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.452035 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istiod-ca-cert\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:20:57.452138 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.452078 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istio-podinfo\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:20:57.452171 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.452142 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-workload-certs\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:20:57.452171 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.452157 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l74qr\" (UniqueName: \"kubernetes.io/projected/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-kube-api-access-l74qr\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:20:57.452232 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.452190 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-workload-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:20:57.512781 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.512744 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk"] Nov 04 01:20:57.517586 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.517562 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:20:57.534615 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.533270 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk"] Nov 04 01:20:57.553299 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.553234 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-workload-certs\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:20:57.553299 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.553272 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l74qr\" (UniqueName: \"kubernetes.io/projected/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-kube-api-access-l74qr\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:20:57.553477 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.553320 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-workload-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:20:57.553538 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.553510 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-credential-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:20:57.553589 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.553566 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istio-token\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:20:57.553645 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.553596 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istio-data\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:20:57.553645 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.553629 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istio-envoy\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:20:57.553748 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.553657 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istiod-ca-cert\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:20:57.553748 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.553676 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-workload-certs\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:20:57.553853 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.553745 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istio-podinfo\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:20:57.553853 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.553814 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-workload-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:20:57.554140 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.554115 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-credential-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:20:57.554234 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.554186 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istio-data\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:20:57.554363 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.554343 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istiod-ca-cert\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:20:57.557094 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.557068 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istio-podinfo\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:20:57.557242 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.557108 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istio-envoy\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:20:57.568225 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.568202 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istio-token\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:20:57.568397 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.568379 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l74qr\" (UniqueName: \"kubernetes.io/projected/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-kube-api-access-l74qr\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:20:57.617539 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.617510 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:20:57.621955 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.621907 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx"] Nov 04 01:20:57.627775 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.627750 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:20:57.655275 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.655073 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:20:57.659149 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.655604 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:20:57.659149 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.655741 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:20:57.659149 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.656128 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:20:57.659149 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.656201 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:20:57.659149 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.656277 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f164affe-f5b7-47dd-a165-2fe6f200caca-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:20:57.659149 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.656315 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f164affe-f5b7-47dd-a165-2fe6f200caca-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:20:57.659149 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.656383 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wl6b\" (UniqueName: \"kubernetes.io/projected/f164affe-f5b7-47dd-a165-2fe6f200caca-kube-api-access-8wl6b\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:20:57.659149 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.656449 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f164affe-f5b7-47dd-a165-2fe6f200caca-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:20:57.672891 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.672869 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx"] Nov 04 01:20:57.757267 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.757234 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f164affe-f5b7-47dd-a165-2fe6f200caca-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:20:57.757523 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.757278 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f164affe-f5b7-47dd-a165-2fe6f200caca-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:20:57.757523 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.757345 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istio-token\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:20:57.757523 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.757379 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-workload-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:20:57.757523 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.757411 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istio-envoy\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:20:57.757523 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.757463 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wl6b\" (UniqueName: \"kubernetes.io/projected/f164affe-f5b7-47dd-a165-2fe6f200caca-kube-api-access-8wl6b\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:20:57.757523 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.757500 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istio-podinfo\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:20:57.757862 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.757526 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istiod-ca-cert\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:20:57.757862 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.757574 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f164affe-f5b7-47dd-a165-2fe6f200caca-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:20:57.757862 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.757645 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:20:57.758169 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.758113 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:20:57.758453 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.758424 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:20:57.758571 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.758468 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-workload-certs\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:20:57.758571 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.758499 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-credential-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:20:57.758571 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.758565 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:20:57.758785 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.758592 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:20:57.758785 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.758643 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj8g4\" (UniqueName: \"kubernetes.io/projected/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-kube-api-access-cj8g4\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:20:57.758785 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.758749 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:20:57.758785 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.758754 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f164affe-f5b7-47dd-a165-2fe6f200caca-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:20:57.758785 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.758777 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istio-data\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:20:57.759269 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.759164 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:20:57.759369 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.759352 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:20:57.759583 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.759538 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:20:57.763624 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.763578 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f164affe-f5b7-47dd-a165-2fe6f200caca-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:20:57.766324 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.766298 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:20:57.767500 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.767475 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f164affe-f5b7-47dd-a165-2fe6f200caca-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:20:57.770532 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.770507 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wl6b\" (UniqueName: \"kubernetes.io/projected/f164affe-f5b7-47dd-a165-2fe6f200caca-kube-api-access-8wl6b\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:20:57.843010 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.842913 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:20:57.860144 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.860117 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istio-token\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:20:57.860255 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.860155 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-workload-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:20:57.860255 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.860184 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istio-envoy\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:20:57.860255 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.860240 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istio-podinfo\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:20:57.860442 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.860263 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istiod-ca-cert\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:20:57.860501 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.860441 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-workload-certs\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:20:57.860501 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.860462 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-credential-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:20:57.860501 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.860490 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj8g4\" (UniqueName: \"kubernetes.io/projected/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-kube-api-access-cj8g4\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:20:57.860653 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.860540 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istio-data\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:20:57.860653 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.860585 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-workload-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:20:57.861411 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.860748 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-workload-certs\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:20:57.861411 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.860894 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istio-data\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:20:57.861411 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.861017 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-credential-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:20:57.861411 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.861332 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istiod-ca-cert\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:20:57.863257 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.863237 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istio-envoy\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:20:57.863771 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.863751 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istio-podinfo\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:20:57.868752 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.868722 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj8g4\" (UniqueName: \"kubernetes.io/projected/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-kube-api-access-cj8g4\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:20:57.868862 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.868798 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istio-token\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:20:57.939680 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.939612 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9"] Nov 04 01:20:57.953648 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.952709 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:20:57.953648 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.952784 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:20:57.953648 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.952816 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:20:57.991560 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:57.991532 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:20:58.133888 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:58.133854 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk"] Nov 04 01:20:58.134776 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:20:58.134743 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf164affe_f5b7_47dd_a165_2fe6f200caca.slice/crio-bfa55e4772eab9ec18cea0a0d7c2618fa7fad717fc0644a3195b442e27b0684e WatchSource:0}: Error finding container bfa55e4772eab9ec18cea0a0d7c2618fa7fad717fc0644a3195b442e27b0684e: Status 404 returned error can't find the container with id bfa55e4772eab9ec18cea0a0d7c2618fa7fad717fc0644a3195b442e27b0684e Nov 04 01:20:58.138131 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:58.138095 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:20:58.138245 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:58.138181 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:20:58.138245 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:58.138224 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:20:58.284362 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:58.284310 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx"] Nov 04 01:20:58.294897 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:58.294862 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:20:58.295332 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:58.294965 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:20:58.295332 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:58.294996 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:20:58.324139 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:58.324107 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" event={"ID":"7f3f33a9-ead2-4fb8-b066-5a45f35881c9","Type":"ContainerStarted","Data":"4c75c8794daef6c19284f9044940dc0b1f6d1566219ac71bb0a768ec8d11031b"} Nov 04 01:20:58.324253 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:58.324168 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" event={"ID":"7f3f33a9-ead2-4fb8-b066-5a45f35881c9","Type":"ContainerStarted","Data":"1f0f6152c5c286efb0b458316c6a6b14b3f27372c2658fa7c87207431177b071"} Nov 04 01:20:58.327599 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:58.325832 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" event={"ID":"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512","Type":"ContainerStarted","Data":"5216a8bbbc45ea0622396944364dc6789d9c5dc482386050a7608b9235c2f6c2"} Nov 04 01:20:58.329855 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:58.329829 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" event={"ID":"f164affe-f5b7-47dd-a165-2fe6f200caca","Type":"ContainerStarted","Data":"18e55ff02e993f9aa000a331180e12e8faf85ecf5c7a40239bacd4c6e1daebc1"} Nov 04 01:20:58.330054 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:58.330033 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" event={"ID":"f164affe-f5b7-47dd-a165-2fe6f200caca","Type":"ContainerStarted","Data":"bfa55e4772eab9ec18cea0a0d7c2618fa7fad717fc0644a3195b442e27b0684e"} Nov 04 01:20:58.348336 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:58.348285 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" podStartSLOduration=1.348269712 podStartE2EDuration="1.348269712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:58.347146958 +0000 UTC m=+1343.288995861" watchObservedRunningTime="2026-11-04 01:20:58.348269712 +0000 UTC m=+1343.290118615" Nov 04 01:20:58.373226 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:58.372639 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" podStartSLOduration=1.372622931 podStartE2EDuration="1.372622931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:58.3689006 +0000 UTC m=+1343.310749502" watchObservedRunningTime="2026-11-04 01:20:58.372622931 +0000 UTC m=+1343.314471814" Nov 04 01:20:58.618054 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:58.618014 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:20:58.848950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:58.845026 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:20:59.414462 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:59.412684 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" event={"ID":"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512","Type":"ContainerStarted","Data":"2fd81e6a3acd8d6f20fd64e0e03d303b1e2988465e961b7b44ebcbfc33c8053f"} Nov 04 01:20:59.478364 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:59.478291 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" podStartSLOduration=2.478267174 podStartE2EDuration="2.478267174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:59.454192546 +0000 UTC m=+1344.396041449" watchObservedRunningTime="2026-11-04 01:20:59.478267174 +0000 UTC m=+1344.420116078" Nov 04 01:20:59.621285 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:59.621253 7059 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 04 01:20:59.621465 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:59.621302 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" podUID="7f3f33a9-ead2-4fb8-b066-5a45f35881c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 04 01:20:59.863949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:59.863093 7059 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 04 01:20:59.863949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:59.863152 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" podUID="f164affe-f5b7-47dd-a165-2fe6f200caca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 04 01:20:59.994949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:20:59.994787 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:21:00.007972 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:00.002076 7059 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 04 01:21:00.007972 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:00.002139 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" podUID="e2652bfd-0f50-4f9b-9bc6-1cd0c9167512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 04 01:21:00.619345 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:00.619305 7059 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 04 01:21:00.619957 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:00.619899 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" podUID="7f3f33a9-ead2-4fb8-b066-5a45f35881c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 04 01:21:00.862585 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:00.862043 7059 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 04 01:21:00.862585 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:00.862115 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" podUID="f164affe-f5b7-47dd-a165-2fe6f200caca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 04 01:21:00.992598 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:00.992132 7059 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 04 01:21:00.992598 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:00.992193 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" podUID="e2652bfd-0f50-4f9b-9bc6-1cd0c9167512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 04 01:21:01.618948 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:01.618533 7059 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 04 01:21:01.618948 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:01.618591 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" podUID="7f3f33a9-ead2-4fb8-b066-5a45f35881c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 04 01:21:01.942949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:01.941159 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:02.030961 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:01.949522 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:21:02.076280 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:02.076238 7059 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 04 01:21:02.305005 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:02.288041 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" podUID="e2652bfd-0f50-4f9b-9bc6-1cd0c9167512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 04 01:21:02.305005 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:02.181249 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 04 01:21:02.305005 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:02.102760 7059 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 04 01:21:02.305005 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:02.288871 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" podUID="f164affe-f5b7-47dd-a165-2fe6f200caca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 04 01:21:02.315950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:02.315480 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:02.649948 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:02.648090 7059 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 04 01:21:02.649948 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:02.648154 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" podUID="7f3f33a9-ead2-4fb8-b066-5a45f35881c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 04 01:21:02.889952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:02.846541 7059 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 04 01:21:02.985950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:02.985261 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" podUID="f164affe-f5b7-47dd-a165-2fe6f200caca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 04 01:21:02.993949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:02.986517 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:03.003492 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:02.994544 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:21:03.017481 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:03.017435 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 04 01:21:03.019147 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:03.019119 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:21:03.020419 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:03.020399 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:21:03.049301 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:03.047347 7059 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 04 01:21:03.051260 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:03.051228 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" podUID="e2652bfd-0f50-4f9b-9bc6-1cd0c9167512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 04 01:21:03.519951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:03.519077 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 04 01:21:03.540952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:03.523946 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 04 01:21:03.629948 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:03.629041 7059 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 04 01:21:03.629948 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:03.629112 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" podUID="7f3f33a9-ead2-4fb8-b066-5a45f35881c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 04 01:21:03.910767 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:03.910670 7059 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 04 01:21:04.002494 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:04.002446 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" podUID="f164affe-f5b7-47dd-a165-2fe6f200caca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 04 01:21:04.003432 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:03.995179 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:04.003868 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:03.997530 7059 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 04 01:21:04.004078 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:04.004052 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" podUID="e2652bfd-0f50-4f9b-9bc6-1cd0c9167512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 04 01:21:04.037982 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:04.037950 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:21:04.085195 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:04.085169 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:21:04.086959 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:04.079209 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:21:04.087995 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:04.079171 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 04 01:21:04.089254 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:04.089236 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:21:04.319953 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:04.272348 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": context deadline exceeded" Nov 04 01:21:04.430950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:04.430853 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:21:04.522113 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:04.519774 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:04.525045 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:04.525009 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:04.714752 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:04.714714 7059 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 04 01:21:04.715065 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:04.715033 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" podUID="7f3f33a9-ead2-4fb8-b066-5a45f35881c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 04 01:21:04.891113 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:04.888453 7059 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 04 01:21:04.891113 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:04.888512 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" podUID="f164affe-f5b7-47dd-a165-2fe6f200caca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 04 01:21:05.017662 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:05.016548 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:05.034402 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:05.034260 7059 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 04 01:21:05.034402 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:05.034338 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" podUID="e2652bfd-0f50-4f9b-9bc6-1cd0c9167512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 04 01:21:05.103062 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:05.103016 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:21:05.531075 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:05.530119 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:05.531075 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:05.530784 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 04 01:21:05.624030 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:05.623034 7059 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 04 01:21:05.624030 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:05.623095 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" podUID="7f3f33a9-ead2-4fb8-b066-5a45f35881c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 04 01:21:05.732209 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:05.732027 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:21:05.733904 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:05.733875 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:21:05.735151 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:05.735131 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:21:05.735347 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:05.735334 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:21:06.026945 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.026890 7059 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 04 01:21:06.029154 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.029118 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" podUID="e2652bfd-0f50-4f9b-9bc6-1cd0c9167512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 04 01:21:06.030069 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.030049 7059 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 04 01:21:06.030245 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.030221 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" podUID="f164affe-f5b7-47dd-a165-2fe6f200caca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 04 01:21:06.237033 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.236976 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72"] Nov 04 01:21:06.357343 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.357265 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:21:06.369949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.366425 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72"] Nov 04 01:21:06.388303 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.388277 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-099dd9\"/\"kube-root-ca.crt\"" Nov 04 01:21:06.438267 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.438236 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-099dd9\"/\"distinct-2-w4-099dd9-kserve-self-signed-certs\"" Nov 04 01:21:06.438597 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.438576 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-099dd9\"/\"default-dockercfg-jnlg9\"" Nov 04 01:21:06.474455 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.474426 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-099dd9\"/\"openshift-service-ca.crt\"" Nov 04 01:21:06.525786 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.525750 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a747fae3-da69-424e-89c2-28fa527999e8-home\") pod \"distinct-2-w4-099dd9-kserve-6ff796867-jdb72\" (UID: \"a747fae3-da69-424e-89c2-28fa527999e8\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:21:06.526085 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.526064 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a747fae3-da69-424e-89c2-28fa527999e8-dshm\") pod \"distinct-2-w4-099dd9-kserve-6ff796867-jdb72\" (UID: \"a747fae3-da69-424e-89c2-28fa527999e8\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:21:06.526281 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.526264 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk4g9\" (UniqueName: \"kubernetes.io/projected/a747fae3-da69-424e-89c2-28fa527999e8-kube-api-access-lk4g9\") pod \"distinct-2-w4-099dd9-kserve-6ff796867-jdb72\" (UID: \"a747fae3-da69-424e-89c2-28fa527999e8\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:21:06.526555 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.526538 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a747fae3-da69-424e-89c2-28fa527999e8-tmp-dir\") pod \"distinct-2-w4-099dd9-kserve-6ff796867-jdb72\" (UID: \"a747fae3-da69-424e-89c2-28fa527999e8\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:21:06.526711 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.526694 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a747fae3-da69-424e-89c2-28fa527999e8-tls-certs\") pod \"distinct-2-w4-099dd9-kserve-6ff796867-jdb72\" (UID: \"a747fae3-da69-424e-89c2-28fa527999e8\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:21:06.526882 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.526858 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a747fae3-da69-424e-89c2-28fa527999e8-model-cache\") pod \"distinct-2-w4-099dd9-kserve-6ff796867-jdb72\" (UID: \"a747fae3-da69-424e-89c2-28fa527999e8\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:21:06.627894 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.627236 7059 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 04 01:21:06.627894 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.627315 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" podUID="7f3f33a9-ead2-4fb8-b066-5a45f35881c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 04 01:21:06.629472 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.629410 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a747fae3-da69-424e-89c2-28fa527999e8-home\") pod \"distinct-2-w4-099dd9-kserve-6ff796867-jdb72\" (UID: \"a747fae3-da69-424e-89c2-28fa527999e8\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:21:06.632330 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.632037 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a747fae3-da69-424e-89c2-28fa527999e8-dshm\") pod \"distinct-2-w4-099dd9-kserve-6ff796867-jdb72\" (UID: \"a747fae3-da69-424e-89c2-28fa527999e8\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:21:06.632330 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.632132 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk4g9\" (UniqueName: \"kubernetes.io/projected/a747fae3-da69-424e-89c2-28fa527999e8-kube-api-access-lk4g9\") pod \"distinct-2-w4-099dd9-kserve-6ff796867-jdb72\" (UID: \"a747fae3-da69-424e-89c2-28fa527999e8\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:21:06.632330 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.632325 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a747fae3-da69-424e-89c2-28fa527999e8-tmp-dir\") pod \"distinct-2-w4-099dd9-kserve-6ff796867-jdb72\" (UID: \"a747fae3-da69-424e-89c2-28fa527999e8\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:21:06.632556 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.632372 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a747fae3-da69-424e-89c2-28fa527999e8-tls-certs\") pod \"distinct-2-w4-099dd9-kserve-6ff796867-jdb72\" (UID: \"a747fae3-da69-424e-89c2-28fa527999e8\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:21:06.632556 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.632409 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a747fae3-da69-424e-89c2-28fa527999e8-model-cache\") pod \"distinct-2-w4-099dd9-kserve-6ff796867-jdb72\" (UID: \"a747fae3-da69-424e-89c2-28fa527999e8\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:21:06.633390 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.632914 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a747fae3-da69-424e-89c2-28fa527999e8-model-cache\") pod \"distinct-2-w4-099dd9-kserve-6ff796867-jdb72\" (UID: \"a747fae3-da69-424e-89c2-28fa527999e8\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:21:06.633390 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.629963 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a747fae3-da69-424e-89c2-28fa527999e8-home\") pod \"distinct-2-w4-099dd9-kserve-6ff796867-jdb72\" (UID: \"a747fae3-da69-424e-89c2-28fa527999e8\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:21:06.633390 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.633217 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a747fae3-da69-424e-89c2-28fa527999e8-tmp-dir\") pod \"distinct-2-w4-099dd9-kserve-6ff796867-jdb72\" (UID: \"a747fae3-da69-424e-89c2-28fa527999e8\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:21:06.681500 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.681439 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a747fae3-da69-424e-89c2-28fa527999e8-tls-certs\") pod \"distinct-2-w4-099dd9-kserve-6ff796867-jdb72\" (UID: \"a747fae3-da69-424e-89c2-28fa527999e8\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:21:06.708989 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.707222 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a747fae3-da69-424e-89c2-28fa527999e8-dshm\") pod \"distinct-2-w4-099dd9-kserve-6ff796867-jdb72\" (UID: \"a747fae3-da69-424e-89c2-28fa527999e8\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:21:06.720354 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:21:06.720196 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:21:06.750042 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.750004 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk4g9\" (UniqueName: \"kubernetes.io/projected/a747fae3-da69-424e-89c2-28fa527999e8-kube-api-access-lk4g9\") pod \"distinct-2-w4-099dd9-kserve-6ff796867-jdb72\" (UID: \"a747fae3-da69-424e-89c2-28fa527999e8\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:21:06.849624 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.849586 7059 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 04 01:21:06.855035 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.854990 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" podUID="f164affe-f5b7-47dd-a165-2fe6f200caca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 04 01:21:06.883622 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.883340 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:21:06.995432 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.995136 7059 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 04 01:21:06.995432 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:06.995196 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" podUID="e2652bfd-0f50-4f9b-9bc6-1cd0c9167512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 04 01:21:07.231184 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:07.231152 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72"] Nov 04 01:21:07.241908 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:21:07.241875 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda747fae3_da69_424e_89c2_28fa527999e8.slice/crio-f8182e7b24dffa06ea65d33be2c554b9ccaa444cbb5fc78e736cb692b72f5c52 WatchSource:0}: Error finding container f8182e7b24dffa06ea65d33be2c554b9ccaa444cbb5fc78e736cb692b72f5c52: Status 404 returned error can't find the container with id f8182e7b24dffa06ea65d33be2c554b9ccaa444cbb5fc78e736cb692b72f5c52 Nov 04 01:21:08.257311 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:08.257265 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" event={"ID":"a747fae3-da69-424e-89c2-28fa527999e8","Type":"ContainerStarted","Data":"99883f48f1c9350ba7609da3c9ef21184e7f88e8ce0945ccb92bc3374480d24b"} Nov 04 01:21:08.257751 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:08.257327 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" event={"ID":"a747fae3-da69-424e-89c2-28fa527999e8","Type":"ContainerStarted","Data":"f8182e7b24dffa06ea65d33be2c554b9ccaa444cbb5fc78e736cb692b72f5c52"} Nov 04 01:21:08.276664 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:08.276599 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podStartSLOduration=2.276578385 podStartE2EDuration="2.276578385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:08.273563404 +0000 UTC m=+1353.215412308" watchObservedRunningTime="2026-11-04 01:21:08.276578385 +0000 UTC m=+1353.218427289" Nov 04 01:21:08.610539 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:08.608986 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:21:08.610539 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:08.610337 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:21:08.611588 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:08.611569 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:21:08.673952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:08.662013 7059 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:08.673952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:08.662070 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" podUID="7f3f33a9-ead2-4fb8-b066-5a45f35881c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:08.884988 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:08.884630 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:21:08.900134 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:08.899315 7059 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:08.900134 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:08.899413 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" podUID="f164affe-f5b7-47dd-a165-2fe6f200caca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:09.007467 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:09.007437 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:21:09.008270 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:09.008245 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:21:09.009217 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:09.009194 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:21:09.517857 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:09.517826 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:21:09.518334 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:09.518186 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:21:09.519152 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:09.519132 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:21:12.180712 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.173473 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44"] Nov 04 01:21:12.180712 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.180152 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:21:12.208857 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.208827 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44"] Nov 04 01:21:12.247140 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.247104 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0085890d-7ef5-4947-975c-440894141023-istiod-ca-cert\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:21:12.247344 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.247169 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0085890d-7ef5-4947-975c-440894141023-istio-token\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:21:12.247447 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.247421 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj6sr\" (UniqueName: \"kubernetes.io/projected/0085890d-7ef5-4947-975c-440894141023-kube-api-access-qj6sr\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:21:12.247510 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.247472 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-workload-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:21:12.247606 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.247588 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-istio-envoy\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:21:12.247683 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.247631 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-credential-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:21:12.247683 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.247655 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-istio-data\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:21:12.247683 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.247669 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0085890d-7ef5-4947-975c-440894141023-istio-podinfo\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:21:12.247845 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.247720 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-workload-certs\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:21:12.348856 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.348823 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj6sr\" (UniqueName: \"kubernetes.io/projected/0085890d-7ef5-4947-975c-440894141023-kube-api-access-qj6sr\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:21:12.349061 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.348875 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-workload-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:21:12.349061 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.348978 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-istio-envoy\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:21:12.349061 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.349008 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-credential-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:21:12.349061 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.349026 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-istio-data\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:21:12.349061 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.349049 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0085890d-7ef5-4947-975c-440894141023-istio-podinfo\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:21:12.349348 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.349094 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-workload-certs\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:21:12.349348 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.349179 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0085890d-7ef5-4947-975c-440894141023-istiod-ca-cert\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:21:12.349348 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.349229 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0085890d-7ef5-4947-975c-440894141023-istio-token\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:21:12.349494 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.349382 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-workload-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:21:12.349494 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.349462 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-credential-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:21:12.349624 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.349565 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-workload-certs\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:21:12.349624 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.349607 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-istio-data\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:21:12.350108 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.350082 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0085890d-7ef5-4947-975c-440894141023-istiod-ca-cert\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:21:12.351780 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.351754 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-istio-envoy\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:21:12.352023 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.352004 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0085890d-7ef5-4947-975c-440894141023-istio-podinfo\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:21:12.367710 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.367685 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj6sr\" (UniqueName: \"kubernetes.io/projected/0085890d-7ef5-4947-975c-440894141023-kube-api-access-qj6sr\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:21:12.369121 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.369101 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0085890d-7ef5-4947-975c-440894141023-istio-token\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:21:12.503680 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.503608 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:21:12.793075 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.793048 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44"] Nov 04 01:21:12.793204 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:21:12.793146 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0085890d_7ef5_4947_975c_440894141023.slice/crio-b53e837e5475edaec76d7d5ed1902f632222bd9f9e239fc3bd2e3d8e773b42db WatchSource:0}: Error finding container b53e837e5475edaec76d7d5ed1902f632222bd9f9e239fc3bd2e3d8e773b42db: Status 404 returned error can't find the container with id b53e837e5475edaec76d7d5ed1902f632222bd9f9e239fc3bd2e3d8e773b42db Nov 04 01:21:12.795750 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.795712 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:21:12.795842 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.795796 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:21:12.795884 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:12.795843 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:21:13.295609 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:13.295573 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" event={"ID":"0085890d-7ef5-4947-975c-440894141023","Type":"ContainerStarted","Data":"d5d64953a1e2923156d28020c73b9da28815ba42cbfa406c02a35d50527fd1fd"} Nov 04 01:21:13.296078 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:13.295629 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" event={"ID":"0085890d-7ef5-4947-975c-440894141023","Type":"ContainerStarted","Data":"b53e837e5475edaec76d7d5ed1902f632222bd9f9e239fc3bd2e3d8e773b42db"} Nov 04 01:21:13.326453 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:13.326407 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" podStartSLOduration=1.3263938579999999 podStartE2EDuration="1.326393858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:13.322405163 +0000 UTC m=+1358.264254066" watchObservedRunningTime="2026-11-04 01:21:13.326393858 +0000 UTC m=+1358.268242760" Nov 04 01:21:13.504635 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:13.504605 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:21:13.506149 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:13.506128 7059 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 04 01:21:13.506251 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:13.506177 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" podUID="0085890d-7ef5-4947-975c-440894141023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 04 01:21:14.504765 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:14.504734 7059 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 04 01:21:14.505169 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:14.504790 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" podUID="0085890d-7ef5-4947-975c-440894141023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 04 01:21:15.272798 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:15.272752 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g"] Nov 04 01:21:15.281953 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:15.279258 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:21:15.292504 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:15.292473 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-499ddd\"/\"distinct-2-w6-499ddd-kserve-self-signed-certs\"" Nov 04 01:21:15.295283 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:15.295248 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg8vv\" (UniqueName: \"kubernetes.io/projected/614e0cfe-0032-4a0c-b5cf-560fb67723c1-kube-api-access-rg8vv\") pod \"distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g\" (UID: \"614e0cfe-0032-4a0c-b5cf-560fb67723c1\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:21:15.296308 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:15.295349 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/614e0cfe-0032-4a0c-b5cf-560fb67723c1-home\") pod \"distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g\" (UID: \"614e0cfe-0032-4a0c-b5cf-560fb67723c1\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:21:15.296308 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:15.295397 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/614e0cfe-0032-4a0c-b5cf-560fb67723c1-tmp-dir\") pod \"distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g\" (UID: \"614e0cfe-0032-4a0c-b5cf-560fb67723c1\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:21:15.296308 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:15.295485 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/614e0cfe-0032-4a0c-b5cf-560fb67723c1-tls-certs\") pod \"distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g\" (UID: \"614e0cfe-0032-4a0c-b5cf-560fb67723c1\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:21:15.296308 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:15.295497 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g"] Nov 04 01:21:15.296308 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:15.295723 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/614e0cfe-0032-4a0c-b5cf-560fb67723c1-dshm\") pod \"distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g\" (UID: \"614e0cfe-0032-4a0c-b5cf-560fb67723c1\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:21:15.296308 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:15.295742 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/614e0cfe-0032-4a0c-b5cf-560fb67723c1-model-cache\") pod \"distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g\" (UID: \"614e0cfe-0032-4a0c-b5cf-560fb67723c1\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:21:15.397597 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:15.397563 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/614e0cfe-0032-4a0c-b5cf-560fb67723c1-dshm\") pod \"distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g\" (UID: \"614e0cfe-0032-4a0c-b5cf-560fb67723c1\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:21:15.397788 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:15.397599 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/614e0cfe-0032-4a0c-b5cf-560fb67723c1-model-cache\") pod \"distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g\" (UID: \"614e0cfe-0032-4a0c-b5cf-560fb67723c1\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:21:15.397788 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:15.397641 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg8vv\" (UniqueName: \"kubernetes.io/projected/614e0cfe-0032-4a0c-b5cf-560fb67723c1-kube-api-access-rg8vv\") pod \"distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g\" (UID: \"614e0cfe-0032-4a0c-b5cf-560fb67723c1\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:21:15.397788 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:15.397720 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/614e0cfe-0032-4a0c-b5cf-560fb67723c1-home\") pod \"distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g\" (UID: \"614e0cfe-0032-4a0c-b5cf-560fb67723c1\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:21:15.397788 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:15.397768 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/614e0cfe-0032-4a0c-b5cf-560fb67723c1-tmp-dir\") pod \"distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g\" (UID: \"614e0cfe-0032-4a0c-b5cf-560fb67723c1\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:21:15.398027 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:15.397888 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/614e0cfe-0032-4a0c-b5cf-560fb67723c1-tls-certs\") pod \"distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g\" (UID: \"614e0cfe-0032-4a0c-b5cf-560fb67723c1\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:21:15.398344 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:15.398316 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/614e0cfe-0032-4a0c-b5cf-560fb67723c1-home\") pod \"distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g\" (UID: \"614e0cfe-0032-4a0c-b5cf-560fb67723c1\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:21:15.398891 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:15.398863 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/614e0cfe-0032-4a0c-b5cf-560fb67723c1-tmp-dir\") pod \"distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g\" (UID: \"614e0cfe-0032-4a0c-b5cf-560fb67723c1\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:21:15.399027 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:15.399003 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/614e0cfe-0032-4a0c-b5cf-560fb67723c1-model-cache\") pod \"distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g\" (UID: \"614e0cfe-0032-4a0c-b5cf-560fb67723c1\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:21:15.400560 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:15.400534 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/614e0cfe-0032-4a0c-b5cf-560fb67723c1-dshm\") pod \"distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g\" (UID: \"614e0cfe-0032-4a0c-b5cf-560fb67723c1\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:21:15.401049 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:15.401030 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/614e0cfe-0032-4a0c-b5cf-560fb67723c1-tls-certs\") pod \"distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g\" (UID: \"614e0cfe-0032-4a0c-b5cf-560fb67723c1\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:21:15.407085 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:15.407063 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg8vv\" (UniqueName: \"kubernetes.io/projected/614e0cfe-0032-4a0c-b5cf-560fb67723c1-kube-api-access-rg8vv\") pod \"distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g\" (UID: \"614e0cfe-0032-4a0c-b5cf-560fb67723c1\") " pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:21:15.505949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:15.505030 7059 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 04 01:21:15.505949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:15.505107 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" podUID="0085890d-7ef5-4947-975c-440894141023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 04 01:21:15.634830 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:15.634756 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:21:15.979951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:15.963591 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g"] Nov 04 01:21:16.363795 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:16.363689 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" event={"ID":"614e0cfe-0032-4a0c-b5cf-560fb67723c1","Type":"ContainerStarted","Data":"973ec1f5711b762177fbe6e7d554dc51906a91e44d4b6f4465dbb38c41a7c5f3"} Nov 04 01:21:16.363795 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:16.363743 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" event={"ID":"614e0cfe-0032-4a0c-b5cf-560fb67723c1","Type":"ContainerStarted","Data":"04e47bb1f21bae48e97d50e5d9a87fe8dd40592fdc3640cb1251597757870cf3"} Nov 04 01:21:16.386990 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:16.386938 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podStartSLOduration=1.3869098530000001 podStartE2EDuration="1.386909853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:16.385826462 +0000 UTC m=+1361.327675365" watchObservedRunningTime="2026-11-04 01:21:16.386909853 +0000 UTC m=+1361.328758811" Nov 04 01:21:16.504638 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:16.504604 7059 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 04 01:21:16.504804 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:16.504672 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" podUID="0085890d-7ef5-4947-975c-440894141023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 04 01:21:16.635712 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:16.635620 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:21:16.884812 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:16.884779 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:21:16.897823 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:16.897751 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:21:17.392074 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:17.392041 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:21:17.505150 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:17.505110 7059 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 04 01:21:17.505516 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:17.505455 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" podUID="0085890d-7ef5-4947-975c-440894141023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 04 01:21:18.504895 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:18.504860 7059 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 04 01:21:18.505295 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:18.504914 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" podUID="0085890d-7ef5-4947-975c-440894141023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 04 01:21:19.668233 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:19.668200 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:21:19.668738 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:19.668610 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:21:19.670034 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:19.670011 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:21:22.752068 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:22.752037 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542"] Nov 04 01:21:22.756957 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:22.756893 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:21:22.771765 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:22.771718 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542"] Nov 04 01:21:22.773276 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:22.773156 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-9c8bf5\"/\"kube-root-ca.crt\"" Nov 04 01:21:22.773505 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:22.773345 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-9c8bf5\"/\"openshift-service-ca.crt\"" Nov 04 01:21:22.773613 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:22.773559 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-9c8bf5\"/\"default-dockercfg-q6h84\"" Nov 04 01:21:22.774847 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:22.774824 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-9c8bf5\"/\"distinct-w5-9c8bf5-kserve-self-signed-certs\"" Nov 04 01:21:22.837251 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:22.837221 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c795fc5-4986-4f4d-b6f4-00baa7738286-dshm\") pod \"distinct-w5-9c8bf5-kserve-5df7696bbc-cg542\" (UID: \"5c795fc5-4986-4f4d-b6f4-00baa7738286\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:21:22.837402 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:22.837376 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c795fc5-4986-4f4d-b6f4-00baa7738286-home\") pod \"distinct-w5-9c8bf5-kserve-5df7696bbc-cg542\" (UID: \"5c795fc5-4986-4f4d-b6f4-00baa7738286\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:21:22.837447 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:22.837431 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c795fc5-4986-4f4d-b6f4-00baa7738286-model-cache\") pod \"distinct-w5-9c8bf5-kserve-5df7696bbc-cg542\" (UID: \"5c795fc5-4986-4f4d-b6f4-00baa7738286\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:21:22.837618 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:22.837594 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c795fc5-4986-4f4d-b6f4-00baa7738286-tmp-dir\") pod \"distinct-w5-9c8bf5-kserve-5df7696bbc-cg542\" (UID: \"5c795fc5-4986-4f4d-b6f4-00baa7738286\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:21:22.837708 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:22.837633 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c795fc5-4986-4f4d-b6f4-00baa7738286-tls-certs\") pod \"distinct-w5-9c8bf5-kserve-5df7696bbc-cg542\" (UID: \"5c795fc5-4986-4f4d-b6f4-00baa7738286\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:21:22.837783 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:22.837766 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wvvb\" (UniqueName: \"kubernetes.io/projected/5c795fc5-4986-4f4d-b6f4-00baa7738286-kube-api-access-4wvvb\") pod \"distinct-w5-9c8bf5-kserve-5df7696bbc-cg542\" (UID: \"5c795fc5-4986-4f4d-b6f4-00baa7738286\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:21:22.950022 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:22.949941 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c795fc5-4986-4f4d-b6f4-00baa7738286-tmp-dir\") pod \"distinct-w5-9c8bf5-kserve-5df7696bbc-cg542\" (UID: \"5c795fc5-4986-4f4d-b6f4-00baa7738286\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:21:22.950022 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:22.950000 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c795fc5-4986-4f4d-b6f4-00baa7738286-tls-certs\") pod \"distinct-w5-9c8bf5-kserve-5df7696bbc-cg542\" (UID: \"5c795fc5-4986-4f4d-b6f4-00baa7738286\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:21:22.950343 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:22.950094 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wvvb\" (UniqueName: \"kubernetes.io/projected/5c795fc5-4986-4f4d-b6f4-00baa7738286-kube-api-access-4wvvb\") pod \"distinct-w5-9c8bf5-kserve-5df7696bbc-cg542\" (UID: \"5c795fc5-4986-4f4d-b6f4-00baa7738286\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:21:22.950343 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:22.950185 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c795fc5-4986-4f4d-b6f4-00baa7738286-dshm\") pod \"distinct-w5-9c8bf5-kserve-5df7696bbc-cg542\" (UID: \"5c795fc5-4986-4f4d-b6f4-00baa7738286\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:21:22.950343 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:22.950321 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c795fc5-4986-4f4d-b6f4-00baa7738286-home\") pod \"distinct-w5-9c8bf5-kserve-5df7696bbc-cg542\" (UID: \"5c795fc5-4986-4f4d-b6f4-00baa7738286\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:21:22.950513 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:22.950389 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c795fc5-4986-4f4d-b6f4-00baa7738286-model-cache\") pod \"distinct-w5-9c8bf5-kserve-5df7696bbc-cg542\" (UID: \"5c795fc5-4986-4f4d-b6f4-00baa7738286\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:21:22.951011 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:22.950899 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c795fc5-4986-4f4d-b6f4-00baa7738286-model-cache\") pod \"distinct-w5-9c8bf5-kserve-5df7696bbc-cg542\" (UID: \"5c795fc5-4986-4f4d-b6f4-00baa7738286\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:21:22.951257 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:22.951215 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c795fc5-4986-4f4d-b6f4-00baa7738286-tmp-dir\") pod \"distinct-w5-9c8bf5-kserve-5df7696bbc-cg542\" (UID: \"5c795fc5-4986-4f4d-b6f4-00baa7738286\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:21:22.952841 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:22.952802 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c795fc5-4986-4f4d-b6f4-00baa7738286-home\") pod \"distinct-w5-9c8bf5-kserve-5df7696bbc-cg542\" (UID: \"5c795fc5-4986-4f4d-b6f4-00baa7738286\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:21:22.959404 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:22.956943 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c795fc5-4986-4f4d-b6f4-00baa7738286-tls-certs\") pod \"distinct-w5-9c8bf5-kserve-5df7696bbc-cg542\" (UID: \"5c795fc5-4986-4f4d-b6f4-00baa7738286\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:21:22.959762 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:22.959740 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c795fc5-4986-4f4d-b6f4-00baa7738286-dshm\") pod \"distinct-w5-9c8bf5-kserve-5df7696bbc-cg542\" (UID: \"5c795fc5-4986-4f4d-b6f4-00baa7738286\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:21:22.966346 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:22.966322 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wvvb\" (UniqueName: \"kubernetes.io/projected/5c795fc5-4986-4f4d-b6f4-00baa7738286-kube-api-access-4wvvb\") pod \"distinct-w5-9c8bf5-kserve-5df7696bbc-cg542\" (UID: \"5c795fc5-4986-4f4d-b6f4-00baa7738286\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:21:23.073602 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:23.073517 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:21:23.359598 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:23.359561 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542"] Nov 04 01:21:23.362555 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:21:23.362529 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c795fc5_4986_4f4d_b6f4_00baa7738286.slice/crio-7de26b13e822c694d2f0881e92a73581c180f03fb4d52bf53e1f2be185201ec7 WatchSource:0}: Error finding container 7de26b13e822c694d2f0881e92a73581c180f03fb4d52bf53e1f2be185201ec7: Status 404 returned error can't find the container with id 7de26b13e822c694d2f0881e92a73581c180f03fb4d52bf53e1f2be185201ec7 Nov 04 01:21:23.429045 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:23.429016 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" event={"ID":"5c795fc5-4986-4f4d-b6f4-00baa7738286","Type":"ContainerStarted","Data":"7de26b13e822c694d2f0881e92a73581c180f03fb4d52bf53e1f2be185201ec7"} Nov 04 01:21:24.438765 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:24.438722 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" event={"ID":"5c795fc5-4986-4f4d-b6f4-00baa7738286","Type":"ContainerStarted","Data":"ad11af4294eb35acf8562e23a44958e42f0a8d839156422c0067125dcd4a1f48"} Nov 04 01:21:24.460804 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:24.460731 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podStartSLOduration=2.460719009 podStartE2EDuration="2.460719009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:24.458057826 +0000 UTC m=+1369.399906744" watchObservedRunningTime="2026-11-04 01:21:24.460719009 +0000 UTC m=+1369.402567953" Nov 04 01:21:25.074287 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:25.074255 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:21:25.635973 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:25.635937 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:21:25.648953 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:25.648913 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:21:26.466446 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:26.466412 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:21:28.394952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:28.383001 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:28.442950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:28.435063 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:28.460949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:28.455186 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:28.527611 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:28.526393 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 04 01:21:28.741949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:28.741012 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:28.741949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:28.741355 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 04 01:21:29.129622 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:29.129524 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:29.384643 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:29.384550 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:29.385259 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:29.385237 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:21:29.425043 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:29.424998 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:29.440085 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:29.440040 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:29.440647 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:29.440623 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:21:29.517107 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:29.474187 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 04 01:21:29.517107 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:29.475704 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:21:29.625950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:29.572679 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:29.625950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:29.604975 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:21:29.768946 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:29.741777 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:29.768946 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:29.763264 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 04 01:21:29.768946 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:29.763563 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:21:29.768946 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:29.763623 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:21:30.130409 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:30.130302 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:30.166953 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:30.149404 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:21:30.256217 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:30.256188 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:21:30.257658 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:30.257450 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:21:30.312346 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:30.312304 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:21:30.340297 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:30.340265 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:21:30.362335 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:30.362304 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:21:30.363233 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:30.363210 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:21:30.383152 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:30.383030 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:21:33.074508 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:33.074475 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:21:33.087747 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:33.087725 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:21:33.964074 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:33.964044 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:21:36.482353 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:21:36.482302 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:21:42.501736 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.501697 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll"] Nov 04 01:21:42.508766 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.508740 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:21:42.519868 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.519844 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll"] Nov 04 01:21:42.657614 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.657583 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:21:42.657814 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.657668 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:21:42.657814 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.657780 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86db204f-72c0-48a4-817d-34067fbf8f0a-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:21:42.657955 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.657833 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hghn\" (UniqueName: \"kubernetes.io/projected/86db204f-72c0-48a4-817d-34067fbf8f0a-kube-api-access-7hghn\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:21:42.657955 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.657900 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86db204f-72c0-48a4-817d-34067fbf8f0a-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:21:42.658070 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.657964 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:21:42.658165 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.658148 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:21:42.658223 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.658190 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86db204f-72c0-48a4-817d-34067fbf8f0a-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:21:42.658314 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.658280 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:21:42.691757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.691728 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn"] Nov 04 01:21:42.697045 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.697024 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:21:42.713067 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.713044 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn"] Nov 04 01:21:42.760271 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.760185 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86db204f-72c0-48a4-817d-34067fbf8f0a-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:21:42.760271 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.760260 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:21:42.760484 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.760352 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:21:42.760484 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.760474 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:21:42.760610 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.760523 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86db204f-72c0-48a4-817d-34067fbf8f0a-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:21:42.760610 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.760564 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hghn\" (UniqueName: \"kubernetes.io/projected/86db204f-72c0-48a4-817d-34067fbf8f0a-kube-api-access-7hghn\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:21:42.760610 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.760600 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86db204f-72c0-48a4-817d-34067fbf8f0a-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:21:42.760776 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.760645 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:21:42.760776 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.760706 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:21:42.760776 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.760746 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:21:42.761092 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.760825 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:21:42.761092 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.760904 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:21:42.761338 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.761175 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:21:42.761680 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.761659 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86db204f-72c0-48a4-817d-34067fbf8f0a-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:21:42.763744 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.763714 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86db204f-72c0-48a4-817d-34067fbf8f0a-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:21:42.763903 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.763878 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:21:42.787071 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.787049 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86db204f-72c0-48a4-817d-34067fbf8f0a-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:21:42.787667 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.787644 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hghn\" (UniqueName: \"kubernetes.io/projected/86db204f-72c0-48a4-817d-34067fbf8f0a-kube-api-access-7hghn\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:21:42.829242 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.829217 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:21:42.863223 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.863185 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-credential-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:21:42.863223 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.863221 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istiod-ca-cert\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:21:42.863412 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.863250 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istio-token\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:21:42.863412 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.863285 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istio-data\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:21:42.863412 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.863328 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-workload-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:21:42.863412 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.863360 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istio-envoy\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:21:42.863412 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.863380 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbmr6\" (UniqueName: \"kubernetes.io/projected/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-kube-api-access-cbmr6\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:21:42.863672 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.863423 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-workload-certs\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:21:42.863672 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.863440 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istio-podinfo\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:21:42.965061 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.965020 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-credential-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:21:42.965061 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.965055 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istiod-ca-cert\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:21:42.965308 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.965097 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istio-token\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:21:42.965308 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.965153 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istio-data\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:21:42.965308 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.965218 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-workload-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:21:42.965308 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.965274 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istio-envoy\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:21:42.965526 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.965312 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbmr6\" (UniqueName: \"kubernetes.io/projected/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-kube-api-access-cbmr6\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:21:42.965526 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.965379 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-workload-certs\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:21:42.965526 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.965405 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istio-podinfo\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:21:42.965526 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.965461 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-credential-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:21:42.965937 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.965891 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istiod-ca-cert\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:21:42.966051 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.966031 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-workload-certs\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:21:42.966282 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.966260 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istio-data\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:21:42.966423 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.966402 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-workload-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:21:42.969159 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.969133 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istio-envoy\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:21:42.969361 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.969336 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istio-podinfo\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:21:42.973697 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.973673 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istio-token\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:21:42.974134 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:42.974113 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbmr6\" (UniqueName: \"kubernetes.io/projected/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-kube-api-access-cbmr6\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:21:43.021968 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:43.021889 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:21:43.134032 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:43.133953 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll"] Nov 04 01:21:43.160905 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:21:43.160867 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86db204f_72c0_48a4_817d_34067fbf8f0a.slice/crio-05b3b63ea597158d88011dadbfb5d9f9d3b577cff3519509145a482bb2bab735 WatchSource:0}: Error finding container 05b3b63ea597158d88011dadbfb5d9f9d3b577cff3519509145a482bb2bab735: Status 404 returned error can't find the container with id 05b3b63ea597158d88011dadbfb5d9f9d3b577cff3519509145a482bb2bab735 Nov 04 01:21:43.163910 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:43.163867 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:21:43.164043 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:43.163988 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:21:43.164090 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:43.164042 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:21:43.311072 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:43.311041 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn"] Nov 04 01:21:43.313591 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:43.313550 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:21:43.313700 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:43.313636 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:21:43.313700 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:43.313677 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:21:44.037426 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:44.037376 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" event={"ID":"7e4a1625-eab6-4e98-b8e8-0ccc12edc542","Type":"ContainerStarted","Data":"f2681d5814ddc28822bd4a6ca61d61351bcc6b7f07b3632583ad4fbc9de41525"} Nov 04 01:21:44.038073 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:44.038044 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" event={"ID":"7e4a1625-eab6-4e98-b8e8-0ccc12edc542","Type":"ContainerStarted","Data":"330de1aa646d646cc6173cc6b8915d3a86b7d65fb11a935f65c3a4522c50c758"} Nov 04 01:21:44.041985 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:44.041952 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" event={"ID":"86db204f-72c0-48a4-817d-34067fbf8f0a","Type":"ContainerStarted","Data":"ac60ec4fb6d14ad97c1911e6248151c036c2f4584abdb302a82d8ce69e8c9b4f"} Nov 04 01:21:44.043317 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:44.043290 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" event={"ID":"86db204f-72c0-48a4-817d-34067fbf8f0a","Type":"ContainerStarted","Data":"05b3b63ea597158d88011dadbfb5d9f9d3b577cff3519509145a482bb2bab735"} Nov 04 01:21:44.078520 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:44.078451 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" podStartSLOduration=2.078431472 podStartE2EDuration="2.078431472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:44.068993333 +0000 UTC m=+1389.010842293" watchObservedRunningTime="2026-11-04 01:21:44.078431472 +0000 UTC m=+1389.020280377" Nov 04 01:21:44.830143 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:44.830104 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:21:44.831783 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:44.831639 7059 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 04 01:21:44.831783 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:44.831691 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" podUID="86db204f-72c0-48a4-817d-34067fbf8f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 04 01:21:45.022984 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:45.022950 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:21:45.024209 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:45.024183 7059 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 04 01:21:45.024311 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:45.024230 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" podUID="7e4a1625-eab6-4e98-b8e8-0ccc12edc542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 04 01:21:45.829516 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:45.829478 7059 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 04 01:21:45.829939 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:45.829531 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" podUID="86db204f-72c0-48a4-817d-34067fbf8f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 04 01:21:46.023245 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:46.023204 7059 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 04 01:21:46.023413 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:46.023257 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" podUID="7e4a1625-eab6-4e98-b8e8-0ccc12edc542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 04 01:21:46.831234 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:46.831198 7059 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 04 01:21:46.831627 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:46.831256 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" podUID="86db204f-72c0-48a4-817d-34067fbf8f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 04 01:21:47.022851 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:47.022815 7059 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 04 01:21:47.023053 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:47.022876 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" podUID="7e4a1625-eab6-4e98-b8e8-0ccc12edc542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 04 01:21:47.834948 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:47.830727 7059 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 04 01:21:47.834948 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:47.830826 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" podUID="86db204f-72c0-48a4-817d-34067fbf8f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 04 01:21:48.022840 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:48.022807 7059 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 04 01:21:48.023039 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:48.022865 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" podUID="7e4a1625-eab6-4e98-b8e8-0ccc12edc542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 04 01:21:48.830279 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:48.830249 7059 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 04 01:21:48.830517 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:48.830495 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" podUID="86db204f-72c0-48a4-817d-34067fbf8f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 04 01:21:49.022979 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:49.022915 7059 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 04 01:21:49.023362 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:49.022995 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" podUID="7e4a1625-eab6-4e98-b8e8-0ccc12edc542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 04 01:21:49.829715 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:49.829679 7059 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 04 01:21:49.829894 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:49.829743 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" podUID="86db204f-72c0-48a4-817d-34067fbf8f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 04 01:21:50.022796 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:50.022755 7059 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 04 01:21:50.023002 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:50.022817 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" podUID="7e4a1625-eab6-4e98-b8e8-0ccc12edc542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 04 01:21:51.895487 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:51.895449 7059 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:51.896186 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:51.896149 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" podUID="86db204f-72c0-48a4-817d-34067fbf8f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:52.027777 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:52.025067 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:52.068826 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:52.068097 7059 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:52.068826 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:52.068180 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" podUID="7e4a1625-eab6-4e98-b8e8-0ccc12edc542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:52.266948 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:52.266471 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:52.278543 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:52.278171 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:52.320111 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:52.320070 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:52.355785 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:52.355743 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 04 01:21:52.367970 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:52.367889 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:52.368841 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:52.368183 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 04 01:21:52.386707 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:52.386676 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:21:52.389192 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:52.386909 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:52.389759 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:52.389443 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:21:52.400160 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:52.400138 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:21:52.462951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:52.456352 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" podStartSLOduration=10.456332667 podStartE2EDuration="10.456332667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:44.11940459 +0000 UTC m=+1389.061253494" watchObservedRunningTime="2026-11-04 01:21:52.456332667 +0000 UTC m=+1397.398181572" Nov 04 01:21:52.840950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:52.824289 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:21:52.859669 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:52.845208 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:21:52.882951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:52.872103 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:21:53.056960 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:53.056865 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:53.062952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:53.062413 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:21:53.333701 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:53.306034 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:53.333701 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:53.307545 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:53.333701 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:53.331409 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:53.379675 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:53.359182 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:53.379675 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:53.369163 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:21:53.379675 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:53.375701 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:53.441503 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:53.441467 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:21:53.452907 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:53.452877 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:21:53.453212 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:53.453194 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:21:53.453401 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:53.453387 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:21:53.453589 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:53.453575 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:21:53.453732 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:53.453719 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:21:54.075972 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:54.075906 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.089085 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:54.089031 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.144316 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:54.144272 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.314067 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:54.314019 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 04 01:21:54.339327 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:54.314019 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.358884 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:54.358842 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.403112 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:54.403060 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.463952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:54.438839 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.464727 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:54.448394 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.465233 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:54.464311 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.673591 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:54.673555 7059 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:54.673914 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:54.673883 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" podUID="e2652bfd-0f50-4f9b-9bc6-1cd0c9167512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.795867 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:54.794570 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq"] Nov 04 01:21:54.827264 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:54.827230 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:21:54.853579 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:54.853506 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ee876a\"/\"distinct-2-w0-ee876a-kserve-self-signed-certs\"" Nov 04 01:21:54.919375 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:54.919333 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64d89f45-696f-4871-8178-566852da13f4-dshm\") pod \"distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq\" (UID: \"64d89f45-696f-4871-8178-566852da13f4\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:21:54.919675 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:54.919649 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64d89f45-696f-4871-8178-566852da13f4-tls-certs\") pod \"distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq\" (UID: \"64d89f45-696f-4871-8178-566852da13f4\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:21:54.919829 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:54.919812 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64d89f45-696f-4871-8178-566852da13f4-tmp-dir\") pod \"distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq\" (UID: \"64d89f45-696f-4871-8178-566852da13f4\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:21:54.920032 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:54.920010 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64d89f45-696f-4871-8178-566852da13f4-home\") pod \"distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq\" (UID: \"64d89f45-696f-4871-8178-566852da13f4\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:21:54.920249 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:54.920231 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64d89f45-696f-4871-8178-566852da13f4-model-cache\") pod \"distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq\" (UID: \"64d89f45-696f-4871-8178-566852da13f4\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:21:54.932005 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:54.920483 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8vsp\" (UniqueName: \"kubernetes.io/projected/64d89f45-696f-4871-8178-566852da13f4-kube-api-access-t8vsp\") pod \"distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq\" (UID: \"64d89f45-696f-4871-8178-566852da13f4\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:21:55.031173 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:54.928503 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq"] Nov 04 01:21:55.080683 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:55.080636 7059 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:55.081286 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:55.081238 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" podUID="7f3f33a9-ead2-4fb8-b066-5a45f35881c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.149574 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:55.149540 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64d89f45-696f-4871-8178-566852da13f4-home\") pod \"distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq\" (UID: \"64d89f45-696f-4871-8178-566852da13f4\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:21:55.149963 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:55.149943 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64d89f45-696f-4871-8178-566852da13f4-model-cache\") pod \"distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq\" (UID: \"64d89f45-696f-4871-8178-566852da13f4\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:21:55.150264 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:55.150247 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8vsp\" (UniqueName: \"kubernetes.io/projected/64d89f45-696f-4871-8178-566852da13f4-kube-api-access-t8vsp\") pod \"distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq\" (UID: \"64d89f45-696f-4871-8178-566852da13f4\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:21:55.150542 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:55.150527 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64d89f45-696f-4871-8178-566852da13f4-dshm\") pod \"distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq\" (UID: \"64d89f45-696f-4871-8178-566852da13f4\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:21:55.150729 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:55.150715 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64d89f45-696f-4871-8178-566852da13f4-tls-certs\") pod \"distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq\" (UID: \"64d89f45-696f-4871-8178-566852da13f4\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:21:55.150863 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:55.150848 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64d89f45-696f-4871-8178-566852da13f4-tmp-dir\") pod \"distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq\" (UID: \"64d89f45-696f-4871-8178-566852da13f4\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:21:55.151772 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:55.151752 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64d89f45-696f-4871-8178-566852da13f4-tmp-dir\") pod \"distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq\" (UID: \"64d89f45-696f-4871-8178-566852da13f4\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:21:55.152279 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:55.152259 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64d89f45-696f-4871-8178-566852da13f4-home\") pod \"distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq\" (UID: \"64d89f45-696f-4871-8178-566852da13f4\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:21:55.152746 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:55.152725 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64d89f45-696f-4871-8178-566852da13f4-model-cache\") pod \"distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq\" (UID: \"64d89f45-696f-4871-8178-566852da13f4\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:21:55.182677 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:55.182571 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.236952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:55.227809 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64d89f45-696f-4871-8178-566852da13f4-dshm\") pod \"distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq\" (UID: \"64d89f45-696f-4871-8178-566852da13f4\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:21:55.236952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:55.233900 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8vsp\" (UniqueName: \"kubernetes.io/projected/64d89f45-696f-4871-8178-566852da13f4-kube-api-access-t8vsp\") pod \"distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq\" (UID: \"64d89f45-696f-4871-8178-566852da13f4\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:21:55.241013 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:55.237524 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64d89f45-696f-4871-8178-566852da13f4-tls-certs\") pod \"distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq\" (UID: \"64d89f45-696f-4871-8178-566852da13f4\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:21:55.366952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:55.359161 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 04 01:21:55.432274 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:55.432123 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.437484 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:55.437403 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 04 01:21:55.437733 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:55.437712 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:21:55.475533 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:55.455880 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.535298 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:55.535254 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 04 01:21:55.536125 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:55.536101 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:21:55.685886 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:55.685856 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:21:55.721098 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:55.721023 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:21:55.721379 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:55.721361 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:21:55.744945 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:55.724254 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:21:55.768678 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:55.762763 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:21:55.805209 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:55.799113 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:21:56.041917 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:56.041888 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq"] Nov 04 01:21:56.042557 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:21:56.042525 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64d89f45_696f_4871_8178_566852da13f4.slice/crio-1edde9ed24f4290fc6db75ef21a9e2c3b733ce6fc6780eeb100e826597971532 WatchSource:0}: Error finding container 1edde9ed24f4290fc6db75ef21a9e2c3b733ce6fc6780eeb100e826597971532: Status 404 returned error can't find the container with id 1edde9ed24f4290fc6db75ef21a9e2c3b733ce6fc6780eeb100e826597971532 Nov 04 01:21:56.799286 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:56.799248 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" event={"ID":"64d89f45-696f-4871-8178-566852da13f4","Type":"ContainerStarted","Data":"c37241846aafa2ba33d0da06870d86efc817f546bee6d0e2513e0fa6d879e77b"} Nov 04 01:21:56.799805 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:56.799780 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" event={"ID":"64d89f45-696f-4871-8178-566852da13f4","Type":"ContainerStarted","Data":"1edde9ed24f4290fc6db75ef21a9e2c3b733ce6fc6780eeb100e826597971532"} Nov 04 01:21:56.820090 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:56.820049 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podStartSLOduration=2.820036203 podStartE2EDuration="2.820036203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:56.816249267 +0000 UTC m=+1401.758098197" watchObservedRunningTime="2026-11-04 01:21:56.820036203 +0000 UTC m=+1401.761885106" Nov 04 01:21:57.436149 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.436115 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t"] Nov 04 01:21:57.440641 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.440619 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:21:57.458993 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.458963 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t"] Nov 04 01:21:57.548985 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.548946 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-credential-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:21:57.549194 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.549100 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-istio-data\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:21:57.549372 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.549341 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d211e9ea-abd4-41d1-8011-e884ad33695f-istio-token\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:21:57.549489 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.549471 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-workload-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:21:57.549579 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.549563 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d211e9ea-abd4-41d1-8011-e884ad33695f-istio-podinfo\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:21:57.549655 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.549629 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-workload-certs\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:21:57.549749 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.549732 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d211e9ea-abd4-41d1-8011-e884ad33695f-istiod-ca-cert\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:21:57.549825 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.549786 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-istio-envoy\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:21:57.549880 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.549849 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tmb6\" (UniqueName: \"kubernetes.io/projected/d211e9ea-abd4-41d1-8011-e884ad33695f-kube-api-access-7tmb6\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:21:57.602760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.602729 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq"] Nov 04 01:21:57.607130 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.607109 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:21:57.607263 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.607205 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:21:57.651201 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.651166 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:21:57.651526 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.651501 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d211e9ea-abd4-41d1-8011-e884ad33695f-istio-token\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:21:57.652000 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.651976 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:21:57.652199 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.652179 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:21:57.652313 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.652209 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:21:57.652313 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.652239 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-workload-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:21:57.652513 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.652483 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:21:57.652646 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.652559 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d211e9ea-abd4-41d1-8011-e884ad33695f-istio-podinfo\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:21:57.652646 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.652592 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-workload-certs\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:21:57.652646 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.652600 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-workload-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:21:57.653014 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.652936 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-workload-certs\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:21:57.654819 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.654763 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:21:57.654946 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.654910 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qznkf\" (UniqueName: \"kubernetes.io/projected/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-kube-api-access-qznkf\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:21:57.655042 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.655018 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq"] Nov 04 01:21:57.655102 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.655053 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d211e9ea-abd4-41d1-8011-e884ad33695f-istiod-ca-cert\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:21:57.655156 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.655123 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:21:57.655217 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.655171 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-istio-envoy\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:21:57.655217 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.655206 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tmb6\" (UniqueName: \"kubernetes.io/projected/d211e9ea-abd4-41d1-8011-e884ad33695f-kube-api-access-7tmb6\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:21:57.655331 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.655250 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:21:57.655385 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.655337 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-credential-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:21:57.655544 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.655492 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-istio-data\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:21:57.655729 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.655701 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d211e9ea-abd4-41d1-8011-e884ad33695f-istiod-ca-cert\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:21:57.655966 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.655904 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-credential-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:21:57.656295 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.656254 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-istio-data\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:21:57.656477 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.656460 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d211e9ea-abd4-41d1-8011-e884ad33695f-istio-podinfo\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:21:57.657957 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.657874 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-istio-envoy\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:21:57.681491 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.681462 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tmb6\" (UniqueName: \"kubernetes.io/projected/d211e9ea-abd4-41d1-8011-e884ad33695f-kube-api-access-7tmb6\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:21:57.681709 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.681690 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d211e9ea-abd4-41d1-8011-e884ad33695f-istio-token\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:21:57.757027 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.756951 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:21:57.757027 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.757021 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:21:57.757193 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.757180 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:21:57.757266 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.757250 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:21:57.757319 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.757306 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:21:57.757354 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.757339 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:21:57.757405 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.757390 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:21:57.757489 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.757466 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:21:57.757546 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.757526 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:21:57.757596 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.757546 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:21:57.757654 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.757602 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qznkf\" (UniqueName: \"kubernetes.io/projected/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-kube-api-access-qznkf\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:21:57.757791 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.757768 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:21:57.757893 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.757871 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:21:57.758020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.757988 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:21:57.759861 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.759816 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:21:57.759861 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.759848 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:21:57.760120 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.760100 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:21:57.792073 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.792048 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qznkf\" (UniqueName: \"kubernetes.io/projected/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-kube-api-access-qznkf\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:21:57.801141 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.801059 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:21:57.921823 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:57.921790 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:21:58.068782 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:58.068749 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t"] Nov 04 01:21:58.069802 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:21:58.069776 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd211e9ea_abd4_41d1_8011_e884ad33695f.slice/crio-5b1f5087e7910a8d7912afeef61069391b716a980ddc9c0089f156d6d73d07cb WatchSource:0}: Error finding container 5b1f5087e7910a8d7912afeef61069391b716a980ddc9c0089f156d6d73d07cb: Status 404 returned error can't find the container with id 5b1f5087e7910a8d7912afeef61069391b716a980ddc9c0089f156d6d73d07cb Nov 04 01:21:58.073060 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:58.073026 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:21:58.073133 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:58.073101 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:21:58.073179 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:58.073130 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:21:58.225102 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:21:58.225075 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf384bd4_f417_42a3_b8a2_cddcbddc8d8b.slice/crio-72ad38a368c6f707a8e9390f62dbc5108e7cede70cfcbada7775f6f7f628465a WatchSource:0}: Error finding container 72ad38a368c6f707a8e9390f62dbc5108e7cede70cfcbada7775f6f7f628465a: Status 404 returned error can't find the container with id 72ad38a368c6f707a8e9390f62dbc5108e7cede70cfcbada7775f6f7f628465a Nov 04 01:21:58.225646 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:58.225617 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq"] Nov 04 01:21:58.228132 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:58.228099 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:21:58.228210 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:58.228164 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:21:58.228210 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:58.228195 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:21:58.825778 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:58.825743 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" event={"ID":"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b","Type":"ContainerStarted","Data":"489d393bc645ccd2293f47d7ecd79b0f7743b38ae03b8d48cc61b4e08eb924a8"} Nov 04 01:21:58.826352 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:58.825804 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" event={"ID":"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b","Type":"ContainerStarted","Data":"72ad38a368c6f707a8e9390f62dbc5108e7cede70cfcbada7775f6f7f628465a"} Nov 04 01:21:58.829496 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:58.829464 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" event={"ID":"d211e9ea-abd4-41d1-8011-e884ad33695f","Type":"ContainerStarted","Data":"cef1124da045bcb53fed0343774e1cbc8920bf4566d132b9753d8575b065ff18"} Nov 04 01:21:58.829711 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:58.829695 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" event={"ID":"d211e9ea-abd4-41d1-8011-e884ad33695f","Type":"ContainerStarted","Data":"5b1f5087e7910a8d7912afeef61069391b716a980ddc9c0089f156d6d73d07cb"} Nov 04 01:21:58.892264 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:58.892208 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" podStartSLOduration=1.892192965 podStartE2EDuration="1.892192965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:58.854483674 +0000 UTC m=+1403.796332577" watchObservedRunningTime="2026-11-04 01:21:58.892192965 +0000 UTC m=+1403.834041867" Nov 04 01:21:58.896848 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:58.896798 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" podStartSLOduration=1.8967846160000001 podStartE2EDuration="1.896784616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:58.886777294 +0000 UTC m=+1403.828626199" watchObservedRunningTime="2026-11-04 01:21:58.896784616 +0000 UTC m=+1403.838633518" Nov 04 01:21:58.923869 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:58.923827 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:21:59.762950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:59.762474 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:21:59.762950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:59.762610 7059 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 04 01:21:59.762950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:59.762657 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" podUID="d211e9ea-abd4-41d1-8011-e884ad33695f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 04 01:21:59.923691 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:59.923657 7059 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 04 01:21:59.924087 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:21:59.923712 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" podUID="bf384bd4-f417-42a3-b8a2-cddcbddc8d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 04 01:22:00.763795 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:00.763284 7059 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 04 01:22:00.763795 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:00.763356 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" podUID="d211e9ea-abd4-41d1-8011-e884ad33695f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 04 01:22:00.924187 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:00.924135 7059 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 04 01:22:00.924685 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:00.924218 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" podUID="bf384bd4-f417-42a3-b8a2-cddcbddc8d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 04 01:22:01.763949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:01.760338 7059 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 04 01:22:01.763949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:01.760403 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" podUID="d211e9ea-abd4-41d1-8011-e884ad33695f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 04 01:22:01.925311 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:01.925266 7059 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 04 01:22:01.925851 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:01.925374 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" podUID="bf384bd4-f417-42a3-b8a2-cddcbddc8d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 04 01:22:02.762175 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:02.762141 7059 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 04 01:22:02.762483 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:02.762431 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" podUID="d211e9ea-abd4-41d1-8011-e884ad33695f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 04 01:22:02.922508 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:02.922482 7059 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 04 01:22:02.922734 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:02.922714 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" podUID="bf384bd4-f417-42a3-b8a2-cddcbddc8d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 04 01:22:03.769951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:03.766874 7059 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 04 01:22:03.769951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:03.767096 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" podUID="d211e9ea-abd4-41d1-8011-e884ad33695f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 04 01:22:03.922371 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:03.922337 7059 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 04 01:22:03.922541 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:03.922388 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" podUID="bf384bd4-f417-42a3-b8a2-cddcbddc8d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 04 01:22:04.760314 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:04.760277 7059 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 04 01:22:04.760499 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:04.760347 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" podUID="d211e9ea-abd4-41d1-8011-e884ad33695f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 04 01:22:04.922560 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:04.922514 7059 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 04 01:22:04.923084 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:04.922608 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" podUID="bf384bd4-f417-42a3-b8a2-cddcbddc8d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 04 01:22:05.349025 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:05.348994 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg"] Nov 04 01:22:05.355519 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:05.355486 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg"] Nov 04 01:22:05.355669 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:05.355600 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:22:05.363981 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:05.362135 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-9c8bf5\"/\"distinct-2-w5-9c8bf5-kserve-self-signed-certs\"" Nov 04 01:22:05.496947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:05.492771 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef88c68d-23be-40b4-87b2-dee154f926aa-home\") pod \"distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg\" (UID: \"ef88c68d-23be-40b4-87b2-dee154f926aa\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:22:05.496947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:05.492829 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef88c68d-23be-40b4-87b2-dee154f926aa-dshm\") pod \"distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg\" (UID: \"ef88c68d-23be-40b4-87b2-dee154f926aa\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:22:05.517947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:05.516507 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef88c68d-23be-40b4-87b2-dee154f926aa-tls-certs\") pod \"distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg\" (UID: \"ef88c68d-23be-40b4-87b2-dee154f926aa\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:22:05.517947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:05.516580 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef88c68d-23be-40b4-87b2-dee154f926aa-model-cache\") pod \"distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg\" (UID: \"ef88c68d-23be-40b4-87b2-dee154f926aa\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:22:05.517947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:05.516621 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef88c68d-23be-40b4-87b2-dee154f926aa-tmp-dir\") pod \"distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg\" (UID: \"ef88c68d-23be-40b4-87b2-dee154f926aa\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:22:05.517947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:05.516668 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pslhh\" (UniqueName: \"kubernetes.io/projected/ef88c68d-23be-40b4-87b2-dee154f926aa-kube-api-access-pslhh\") pod \"distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg\" (UID: \"ef88c68d-23be-40b4-87b2-dee154f926aa\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:22:05.609704 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:05.607766 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:22:05.621419 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:05.621380 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef88c68d-23be-40b4-87b2-dee154f926aa-model-cache\") pod \"distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg\" (UID: \"ef88c68d-23be-40b4-87b2-dee154f926aa\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:22:05.621585 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:05.621445 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef88c68d-23be-40b4-87b2-dee154f926aa-tmp-dir\") pod \"distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg\" (UID: \"ef88c68d-23be-40b4-87b2-dee154f926aa\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:22:05.621585 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:05.621492 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pslhh\" (UniqueName: \"kubernetes.io/projected/ef88c68d-23be-40b4-87b2-dee154f926aa-kube-api-access-pslhh\") pod \"distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg\" (UID: \"ef88c68d-23be-40b4-87b2-dee154f926aa\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:22:05.621699 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:05.621652 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef88c68d-23be-40b4-87b2-dee154f926aa-home\") pod \"distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg\" (UID: \"ef88c68d-23be-40b4-87b2-dee154f926aa\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:22:05.621753 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:05.621707 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef88c68d-23be-40b4-87b2-dee154f926aa-dshm\") pod \"distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg\" (UID: \"ef88c68d-23be-40b4-87b2-dee154f926aa\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:22:05.621996 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:05.621977 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef88c68d-23be-40b4-87b2-dee154f926aa-tls-certs\") pod \"distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg\" (UID: \"ef88c68d-23be-40b4-87b2-dee154f926aa\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:22:05.622908 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:05.622879 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef88c68d-23be-40b4-87b2-dee154f926aa-home\") pod \"distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg\" (UID: \"ef88c68d-23be-40b4-87b2-dee154f926aa\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:22:05.623418 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:05.623394 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef88c68d-23be-40b4-87b2-dee154f926aa-model-cache\") pod \"distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg\" (UID: \"ef88c68d-23be-40b4-87b2-dee154f926aa\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:22:05.623550 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:05.623531 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef88c68d-23be-40b4-87b2-dee154f926aa-tmp-dir\") pod \"distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg\" (UID: \"ef88c68d-23be-40b4-87b2-dee154f926aa\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:22:05.626574 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:05.626553 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:22:05.629960 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:05.628168 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef88c68d-23be-40b4-87b2-dee154f926aa-tls-certs\") pod \"distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg\" (UID: \"ef88c68d-23be-40b4-87b2-dee154f926aa\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:22:05.629960 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:05.628236 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef88c68d-23be-40b4-87b2-dee154f926aa-dshm\") pod \"distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg\" (UID: \"ef88c68d-23be-40b4-87b2-dee154f926aa\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:22:05.633557 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:05.633530 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pslhh\" (UniqueName: \"kubernetes.io/projected/ef88c68d-23be-40b4-87b2-dee154f926aa-kube-api-access-pslhh\") pod \"distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg\" (UID: \"ef88c68d-23be-40b4-87b2-dee154f926aa\") " pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:22:05.690751 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:05.690718 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:22:06.048238 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:06.047525 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:22:06.062357 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:06.062320 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg"] Nov 04 01:22:06.192161 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:06.192129 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:22:06.193086 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:06.193056 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:22:06.193955 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:06.193903 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:22:06.292399 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:06.292285 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:22:06.293728 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:06.293512 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:22:06.295436 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:06.295402 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:22:06.514582 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:22:06.514547 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:22:07.016071 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:07.016022 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" event={"ID":"ef88c68d-23be-40b4-87b2-dee154f926aa","Type":"ContainerStarted","Data":"0fc15d0302967ac7fde2be4acb9d9be123221cdcbd0f42b6fb8127b19bc6df2d"} Nov 04 01:22:07.016071 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:07.016076 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" event={"ID":"ef88c68d-23be-40b4-87b2-dee154f926aa","Type":"ContainerStarted","Data":"028b4f0fc428b3edbd6da5b6ec0a2aef670c68305b56e18136ed1b4027ebaf55"} Nov 04 01:22:07.037227 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:07.037183 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podStartSLOduration=2.037169548 podStartE2EDuration="2.037169548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:07.033483752 +0000 UTC m=+1411.975332655" watchObservedRunningTime="2026-11-04 01:22:07.037169548 +0000 UTC m=+1411.979018452" Nov 04 01:22:07.691908 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:07.691874 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:22:12.667033 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:12.666993 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62"] Nov 04 01:22:12.675514 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:12.675479 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:22:12.699557 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:12.699525 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62"] Nov 04 01:22:12.790144 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:12.790105 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istio-token\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:22:12.790294 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:12.790238 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istio-podinfo\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:22:12.790294 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:12.790276 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istio-envoy\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:22:12.790416 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:12.790312 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-workload-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:22:12.790416 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:12.790339 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-workload-certs\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:22:12.790483 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:12.790433 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istiod-ca-cert\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:22:12.790568 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:12.790553 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-credential-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:22:12.790609 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:12.790596 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxnrr\" (UniqueName: \"kubernetes.io/projected/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-kube-api-access-sxnrr\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:22:12.790643 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:12.790620 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istio-data\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:22:12.892483 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:12.892452 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-credential-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:22:12.892671 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:12.892501 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxnrr\" (UniqueName: \"kubernetes.io/projected/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-kube-api-access-sxnrr\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:22:12.892671 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:12.892526 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istio-data\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:22:12.892671 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:12.892626 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istio-token\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:22:12.892898 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:12.892706 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istio-podinfo\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:22:12.892898 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:12.892745 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istio-envoy\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:22:12.892898 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:12.892795 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-workload-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:22:12.892898 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:12.892832 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-workload-certs\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:22:12.892898 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:12.892861 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istiod-ca-cert\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:22:12.893188 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:12.892915 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-credential-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:22:12.893188 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:12.892957 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istio-data\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:22:12.893337 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:12.893312 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-workload-certs\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:22:12.893527 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:12.893501 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-workload-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:22:12.893876 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:12.893853 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istiod-ca-cert\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:22:12.895583 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:12.895556 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istio-envoy\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:22:12.895796 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:12.895771 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istio-podinfo\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:22:12.901840 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:12.901817 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxnrr\" (UniqueName: \"kubernetes.io/projected/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-kube-api-access-sxnrr\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:22:12.902519 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:12.902492 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istio-token\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:22:13.015316 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:13.015286 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:22:13.322857 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:13.322826 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62"] Nov 04 01:22:13.323035 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:22:13.322908 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60fe9f56_ebb0_4b4c_b325_8f3cb73c029c.slice/crio-9ea974a1aa007a2033b0a205a4edd299886841e1396282bf13c0478a4624410d WatchSource:0}: Error finding container 9ea974a1aa007a2033b0a205a4edd299886841e1396282bf13c0478a4624410d: Status 404 returned error can't find the container with id 9ea974a1aa007a2033b0a205a4edd299886841e1396282bf13c0478a4624410d Nov 04 01:22:13.325787 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:13.325742 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:22:13.325890 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:13.325832 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:22:13.325952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:13.325918 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:22:14.076734 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:14.076696 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" event={"ID":"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c","Type":"ContainerStarted","Data":"54b2d67c46b6afe96124bc1cca8e4ad7a996323280e531b3967a482cba02bdd1"} Nov 04 01:22:14.077276 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:14.076765 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" event={"ID":"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c","Type":"ContainerStarted","Data":"9ea974a1aa007a2033b0a205a4edd299886841e1396282bf13c0478a4624410d"} Nov 04 01:22:14.112738 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:14.112667 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" podStartSLOduration=2.112643511 podStartE2EDuration="2.112643511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:14.10182478 +0000 UTC m=+1419.043673708" watchObservedRunningTime="2026-11-04 01:22:14.112643511 +0000 UTC m=+1419.054492419" Nov 04 01:22:15.018563 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:15.015441 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:22:15.018563 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:15.017887 7059 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 04 01:22:15.018563 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:15.017964 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" podUID="60fe9f56-ebb0-4b4c-b325-8f3cb73c029c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 04 01:22:15.691107 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:15.691070 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:22:15.704289 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:15.704263 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:22:16.016352 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:16.016257 7059 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 04 01:22:16.016352 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:16.016321 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" podUID="60fe9f56-ebb0-4b4c-b325-8f3cb73c029c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 04 01:22:16.114197 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:16.114168 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:22:17.016160 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:17.016126 7059 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 04 01:22:17.016587 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:17.016182 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" podUID="60fe9f56-ebb0-4b4c-b325-8f3cb73c029c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 04 01:22:18.016018 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:18.015974 7059 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 04 01:22:18.016188 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:18.016052 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" podUID="60fe9f56-ebb0-4b4c-b325-8f3cb73c029c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 04 01:22:19.015786 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:19.015750 7059 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 04 01:22:19.015970 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:19.015812 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" podUID="60fe9f56-ebb0-4b4c-b325-8f3cb73c029c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 04 01:22:20.016053 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:20.016013 7059 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 04 01:22:20.016464 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:20.016092 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" podUID="60fe9f56-ebb0-4b4c-b325-8f3cb73c029c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 04 01:22:21.019940 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:21.019899 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:22:21.020375 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:21.020201 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:22:21.021053 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:21.021034 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:22:26.014949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:26.014291 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:26.084952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:26.084190 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:26.112395 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:26.112350 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:26.233058 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:26.233010 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:26.388308 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:26.388208 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:26.473309 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:26.473260 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:26.617067 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:26.617011 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:26.653006 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:26.650022 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:26.688081 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:26.688016 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:26.700963 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:26.697427 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:26.727719 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:26.727678 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 04 01:22:26.728410 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:26.728378 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:26.804138 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:26.803018 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:27.046951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:27.029721 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Nov 04 01:22:27.063951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:27.048489 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:22:27.090124 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:27.090076 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:27.135952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:27.094460 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:22:27.234951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:27.120349 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:27.269973 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:27.256178 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:27.442983 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:27.417843 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:27.552960 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:27.539596 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:27.692500 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:27.692451 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:27.728676 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:27.728558 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 04 01:22:27.745557 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:27.745515 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:27.812708 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:27.812666 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:27.916299 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:27.916259 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:27.940969 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:27.939073 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:22:27.940969 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:27.939170 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:22:27.940969 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:27.939245 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:22:27.940969 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:27.939304 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:22:27.940969 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:27.939364 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:22:27.940969 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:27.939419 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:22:27.940969 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:27.939474 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:22:27.940969 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:27.939533 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:22:28.118600 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:28.118487 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:28.156579 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:28.156530 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:28.376655 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:28.376556 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:22:28.382175 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:28.382136 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:28.676095 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:28.659358 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 04 01:22:28.703426 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:28.703373 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:28.730473 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:28.730413 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:28.771194 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:28.771146 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 04 01:22:28.867952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:28.834342 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 04 01:22:28.867952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:28.834706 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 04 01:22:29.166010 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:29.148352 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:29.264373 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:29.264327 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:29.400474 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:29.395690 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:29.406435 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:29.395735 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:29.516967 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:29.483173 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:29.727952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:29.670034 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:30.030227 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:29.759159 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:30.030888 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:29.759196 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 04 01:22:30.031422 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:29.787556 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 04 01:22:30.031824 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:29.840639 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:30.032300 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:29.840676 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:30.156148 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:30.156104 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:30.304643 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:30.304537 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:30.349975 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:22:30.333854 7059 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.162s" Nov 04 01:22:30.412969 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:30.400821 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:30.412969 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:30.409333 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:22:30.412969 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:30.409667 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded" Nov 04 01:22:30.689118 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:30.662101 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:22:30.689118 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:30.663117 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:22:30.689118 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:30.664077 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:30.689118 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:30.687360 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:22:30.800271 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:30.791544 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:22:30.903725 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:30.903697 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:22:30.907120 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:30.907096 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:22:30.908708 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:30.908667 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:22:30.909571 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:30.909552 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:22:30.961214 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:30.961126 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:22:30.986120 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:30.979059 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:22:36.483442 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:22:36.483399 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:22:42.780786 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:42.780745 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4"] Nov 04 01:22:42.792987 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:42.792957 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:22:42.813714 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:42.813665 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4"] Nov 04 01:22:42.826509 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:42.826223 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c"] Nov 04 01:22:42.837612 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:42.837585 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:22:42.852710 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:42.852678 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c"] Nov 04 01:22:42.869796 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:42.869755 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499"] Nov 04 01:22:42.876082 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:42.876042 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:22:42.903642 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:42.903611 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499"] Nov 04 01:22:42.938630 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:42.938599 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9"] Nov 04 01:22:42.943219 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:42.943196 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:22:42.951568 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:42.951432 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:22:42.951568 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:42.951517 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-workload-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:22:42.951568 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:42.951553 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-credential-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:22:42.951819 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:42.951603 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7nkg\" (UniqueName: \"kubernetes.io/projected/cce88e25-4d81-4b48-a229-b72e308c5f18-kube-api-access-l7nkg\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:22:42.951819 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:42.951632 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj9ft\" (UniqueName: \"kubernetes.io/projected/19a17fd1-42d6-473d-9568-ad6ba68ca22a-kube-api-access-kj9ft\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:22:42.951819 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:42.951681 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cce88e25-4d81-4b48-a229-b72e308c5f18-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:22:42.952031 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:42.951846 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:22:42.952031 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:42.951875 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istio-podinfo\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:22:42.952031 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:42.951915 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istiod-ca-cert\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:22:42.952031 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:42.951973 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:22:42.953462 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:42.952080 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce88e25-4d81-4b48-a229-b72e308c5f18-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:22:42.953462 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:42.952158 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce88e25-4d81-4b48-a229-b72e308c5f18-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:22:42.953462 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:42.952215 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:22:42.953462 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:42.952255 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istio-data\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:22:42.953462 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:42.952308 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istio-token\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:22:42.953462 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:42.952339 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:22:42.953462 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:42.952374 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-workload-certs\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:22:42.953462 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:42.952400 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istio-envoy\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:22:42.962887 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:42.962861 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9"] Nov 04 01:22:43.053813 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.053732 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-workload-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:22:43.053813 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.053773 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-credential-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:22:43.054065 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.053815 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7nkg\" (UniqueName: \"kubernetes.io/projected/cce88e25-4d81-4b48-a229-b72e308c5f18-kube-api-access-l7nkg\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:22:43.054065 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.053898 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj9ft\" (UniqueName: \"kubernetes.io/projected/19a17fd1-42d6-473d-9568-ad6ba68ca22a-kube-api-access-kj9ft\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:22:43.054065 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.053989 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:22:43.054065 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.054026 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:22:43.054242 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.054101 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cce88e25-4d81-4b48-a229-b72e308c5f18-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:22:43.054242 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.054131 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:22:43.054242 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.054166 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:22:43.054242 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.054175 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-workload-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:22:43.054242 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.054193 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istio-podinfo\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:22:43.054242 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.054217 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-credential-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:22:43.054242 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.054242 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istiod-ca-cert\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:22:43.054573 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.054297 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:22:43.054573 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.054332 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm9qp\" (UniqueName: \"kubernetes.io/projected/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-kube-api-access-vm9qp\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:22:43.054573 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.054395 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:22:43.054573 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.054433 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:22:43.054573 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.054473 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:22:43.054573 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.054482 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:22:43.054850 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.054653 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eedbad81-5540-45d7-adbb-54afd75aa6e8-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:22:43.054850 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.054738 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:22:43.054850 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.054780 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eedbad81-5540-45d7-adbb-54afd75aa6e8-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:22:43.054850 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.054800 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht2ln\" (UniqueName: \"kubernetes.io/projected/eedbad81-5540-45d7-adbb-54afd75aa6e8-kube-api-access-ht2ln\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:22:43.054850 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.054822 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce88e25-4d81-4b48-a229-b72e308c5f18-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:22:43.054850 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.054850 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:22:43.055092 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.054950 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eedbad81-5540-45d7-adbb-54afd75aa6e8-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:22:43.055092 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.054964 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istiod-ca-cert\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:22:43.055092 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.054976 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:22:43.055092 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.055073 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce88e25-4d81-4b48-a229-b72e308c5f18-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:22:43.055245 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.055188 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:22:43.055245 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.055219 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:22:43.055346 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.055250 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:22:43.055346 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.055320 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istio-data\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:22:43.055456 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.055381 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:22:43.055456 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.055430 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istio-token\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:22:43.055568 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.055552 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:22:43.055621 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.055576 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce88e25-4d81-4b48-a229-b72e308c5f18-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:22:43.055684 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.055619 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:22:43.055684 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.055653 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istio-data\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:22:43.055684 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.055658 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-workload-certs\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:22:43.055840 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.055699 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istio-envoy\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:22:43.055840 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.055717 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:22:43.055840 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.055740 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:22:43.056025 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.055860 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:22:43.056025 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.055943 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:22:43.056025 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.055985 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-workload-certs\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:22:43.057629 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.057599 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istio-podinfo\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:22:43.058167 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.058144 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce88e25-4d81-4b48-a229-b72e308c5f18-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:22:43.058267 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.058213 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istio-envoy\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:22:43.058458 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.058437 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:22:43.063128 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.063099 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7nkg\" (UniqueName: \"kubernetes.io/projected/cce88e25-4d81-4b48-a229-b72e308c5f18-kube-api-access-l7nkg\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:22:43.063243 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.063219 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj9ft\" (UniqueName: \"kubernetes.io/projected/19a17fd1-42d6-473d-9568-ad6ba68ca22a-kube-api-access-kj9ft\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:22:43.063397 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.063372 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istio-token\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:22:43.063457 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.063437 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cce88e25-4d81-4b48-a229-b72e308c5f18-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:22:43.139815 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.139788 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:22:43.156735 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.156711 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:22:43.156828 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.156753 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:22:43.156828 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.156804 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:22:43.156828 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.156821 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm9qp\" (UniqueName: \"kubernetes.io/projected/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-kube-api-access-vm9qp\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:22:43.156989 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.156839 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:22:43.156989 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.156879 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:22:43.156989 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.156964 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eedbad81-5540-45d7-adbb-54afd75aa6e8-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:22:43.157166 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.157040 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eedbad81-5540-45d7-adbb-54afd75aa6e8-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:22:43.157166 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.157068 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht2ln\" (UniqueName: \"kubernetes.io/projected/eedbad81-5540-45d7-adbb-54afd75aa6e8-kube-api-access-ht2ln\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:22:43.157166 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.157111 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:22:43.157337 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.157163 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eedbad81-5540-45d7-adbb-54afd75aa6e8-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:22:43.157337 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.157176 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:22:43.157337 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.157191 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:22:43.157337 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.157210 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:22:43.157337 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.157294 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:22:43.157337 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.157334 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:22:43.157660 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.157394 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:22:43.157660 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.157447 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:22:43.157660 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.157472 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:22:43.157660 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.157490 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:22:43.157660 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.157474 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:22:43.157979 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.157682 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:22:43.158123 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.158051 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:22:43.158227 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.158148 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eedbad81-5540-45d7-adbb-54afd75aa6e8-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:22:43.158377 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.158351 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:22:43.159168 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.158799 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:22:43.159168 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.159131 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:22:43.159331 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.159268 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:22:43.160327 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.160299 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eedbad81-5540-45d7-adbb-54afd75aa6e8-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:22:43.160589 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.160571 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:22:43.161414 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.161389 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:22:43.161834 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.161814 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:22:43.167356 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.167311 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:22:43.167674 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.167648 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht2ln\" (UniqueName: \"kubernetes.io/projected/eedbad81-5540-45d7-adbb-54afd75aa6e8-kube-api-access-ht2ln\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:22:43.167822 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.167734 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eedbad81-5540-45d7-adbb-54afd75aa6e8-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:22:43.169781 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.169593 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:22:43.170024 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.170002 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm9qp\" (UniqueName: \"kubernetes.io/projected/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-kube-api-access-vm9qp\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:22:43.197099 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.197068 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:22:43.268736 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.268694 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:22:43.456783 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.456686 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4"] Nov 04 01:22:43.461301 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.461233 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:22:43.461572 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.461548 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:22:43.461765 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.461730 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:22:43.509764 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.509712 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c"] Nov 04 01:22:43.517426 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:22:43.515620 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a17fd1_42d6_473d_9568_ad6ba68ca22a.slice/crio-297dcccedcf1e2c5de11f28d6c8cffb0b5455fbde343b577706866e931ada778 WatchSource:0}: Error finding container 297dcccedcf1e2c5de11f28d6c8cffb0b5455fbde343b577706866e931ada778: Status 404 returned error can't find the container with id 297dcccedcf1e2c5de11f28d6c8cffb0b5455fbde343b577706866e931ada778 Nov 04 01:22:43.519700 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.519664 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:22:43.519801 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.519736 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:22:43.519801 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.519783 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:22:43.543613 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.543584 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499"] Nov 04 01:22:43.548436 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.548048 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:22:43.548436 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.548116 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:22:43.548436 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.548144 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:22:43.586487 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.586463 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9"] Nov 04 01:22:43.586709 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:22:43.586689 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeedbad81_5540_45d7_adbb_54afd75aa6e8.slice/crio-9336975a2bc1f3c200335a382675672d9f30877ff3cce848e9906e3c7b8a0cef WatchSource:0}: Error finding container 9336975a2bc1f3c200335a382675672d9f30877ff3cce848e9906e3c7b8a0cef: Status 404 returned error can't find the container with id 9336975a2bc1f3c200335a382675672d9f30877ff3cce848e9906e3c7b8a0cef Nov 04 01:22:43.590163 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.590130 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:22:43.590253 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.590191 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:22:43.590253 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.590219 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:22:43.735950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.735902 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" event={"ID":"eedbad81-5540-45d7-adbb-54afd75aa6e8","Type":"ContainerStarted","Data":"9336975a2bc1f3c200335a382675672d9f30877ff3cce848e9906e3c7b8a0cef"} Nov 04 01:22:43.740171 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.740136 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" event={"ID":"b42e3b72-1c9f-4a14-88c8-755b3faceb5a","Type":"ContainerStarted","Data":"090fb0a7cd2fd554cac2320d82f5c4dda1a6fe7de2c781dcf90517fb71d38819"} Nov 04 01:22:43.740546 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.740198 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" event={"ID":"b42e3b72-1c9f-4a14-88c8-755b3faceb5a","Type":"ContainerStarted","Data":"81dcabfb9875cbd94da7aff38f26fe46feee071639d1889da41a6ab209ce2b5d"} Nov 04 01:22:43.744279 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.744244 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" event={"ID":"19a17fd1-42d6-473d-9568-ad6ba68ca22a","Type":"ContainerStarted","Data":"8839dc00f2ba41a9fece0559748d6b6bababd2f2a5f4032adc24a555f48c5c51"} Nov 04 01:22:43.744381 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.744314 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" event={"ID":"19a17fd1-42d6-473d-9568-ad6ba68ca22a","Type":"ContainerStarted","Data":"297dcccedcf1e2c5de11f28d6c8cffb0b5455fbde343b577706866e931ada778"} Nov 04 01:22:43.754311 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.754261 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" event={"ID":"cce88e25-4d81-4b48-a229-b72e308c5f18","Type":"ContainerStarted","Data":"1b14d352d212126128445d15c69a47096833a482797a15d75416be4c251483c3"} Nov 04 01:22:43.754575 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.754543 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" event={"ID":"cce88e25-4d81-4b48-a229-b72e308c5f18","Type":"ContainerStarted","Data":"e78517b3f60621c83ee99c992d551c04956e574d7eac747847611a6d79cf09c2"} Nov 04 01:22:43.767559 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.767378 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" podStartSLOduration=1.767365147 podStartE2EDuration="1.767365147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:43.765236881 +0000 UTC m=+1448.707085785" watchObservedRunningTime="2026-11-04 01:22:43.767365147 +0000 UTC m=+1448.709214049" Nov 04 01:22:43.795310 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.795257 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" podStartSLOduration=1.7952438819999998 podStartE2EDuration="1.795243882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:43.794002472 +0000 UTC m=+1448.735851378" watchObservedRunningTime="2026-11-04 01:22:43.795243882 +0000 UTC m=+1448.737092784" Nov 04 01:22:43.827104 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:43.826989 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" podStartSLOduration=1.82697116 podStartE2EDuration="1.82697116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:43.819705867 +0000 UTC m=+1448.761554771" watchObservedRunningTime="2026-11-04 01:22:43.82697116 +0000 UTC m=+1448.768820066" Nov 04 01:22:44.140632 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:44.140544 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:22:44.141845 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:44.141809 7059 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 04 01:22:44.141990 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:44.141863 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" podUID="cce88e25-4d81-4b48-a229-b72e308c5f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 04 01:22:44.170176 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:44.170149 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:22:44.171725 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:44.171701 7059 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:22:44.171829 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:44.171749 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" podUID="19a17fd1-42d6-473d-9568-ad6ba68ca22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 04 01:22:44.198287 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:44.198258 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:22:44.199359 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:44.199340 7059 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:22:44.199440 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:44.199382 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" podUID="b42e3b72-1c9f-4a14-88c8-755b3faceb5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 04 01:22:45.087764 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:45.087672 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" event={"ID":"eedbad81-5540-45d7-adbb-54afd75aa6e8","Type":"ContainerStarted","Data":"a3d4097794e797d260b9122d55862b113ff5987deb7606389f7dd8d59ce065d1"} Nov 04 01:22:45.142741 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:45.142699 7059 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 04 01:22:45.168950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:45.166299 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" podUID="cce88e25-4d81-4b48-a229-b72e308c5f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 04 01:22:45.194767 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:45.178350 7059 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:22:45.194767 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:45.178424 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" podUID="19a17fd1-42d6-473d-9568-ad6ba68ca22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 04 01:22:45.194767 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:45.186754 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" podStartSLOduration=3.186733042 podStartE2EDuration="3.186733042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:45.14734945 +0000 UTC m=+1450.089198354" watchObservedRunningTime="2026-11-04 01:22:45.186733042 +0000 UTC m=+1450.128581946" Nov 04 01:22:45.280952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:45.280072 7059 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:22:45.280952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:45.280174 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" podUID="b42e3b72-1c9f-4a14-88c8-755b3faceb5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 04 01:22:45.336021 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:45.335051 7059 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 04 01:22:45.336021 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:45.335135 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" podUID="eedbad81-5540-45d7-adbb-54afd75aa6e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 04 01:22:45.488068 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:45.488031 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:22:45.676961 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:45.665056 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:45.676961 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:45.672400 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:45.891360 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:45.891257 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:45.946950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:45.922325 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 04 01:22:46.013950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:46.013363 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:46.058404 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:46.058360 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:46.122744 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:46.122024 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Nov 04 01:22:46.146194 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:46.146107 7059 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 04 01:22:46.146656 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:46.146624 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" podUID="cce88e25-4d81-4b48-a229-b72e308c5f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 04 01:22:46.159946 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:46.156018 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:46.195950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:46.179661 7059 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:22:46.195950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:46.179714 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" podUID="19a17fd1-42d6-473d-9568-ad6ba68ca22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 04 01:22:46.198890 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:46.198783 7059 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:22:46.198890 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:46.198836 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" podUID="b42e3b72-1c9f-4a14-88c8-755b3faceb5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 04 01:22:46.281950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:46.279330 7059 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 04 01:22:46.281950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:46.279390 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" podUID="eedbad81-5540-45d7-adbb-54afd75aa6e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 04 01:22:46.425358 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:46.425320 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:46.426712 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:46.426681 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:46.427204 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:46.426886 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:46.622948 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:46.619013 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": context deadline exceeded" Nov 04 01:22:46.650955 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:46.647224 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:46.676149 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:46.676008 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 04 01:22:46.680096 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:46.676815 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:22:46.723186 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:46.723146 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:47.028007 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:47.027251 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 04 01:22:47.057950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:47.040376 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 04 01:22:47.057950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:47.041809 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:22:47.057950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:47.041868 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:22:47.057950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:47.043434 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:47.057950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:47.045812 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:22:47.066246 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:47.066043 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 04 01:22:47.066658 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:47.066623 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:22:47.501949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:47.434170 7059 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:22:47.525216 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:47.520734 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" podUID="19a17fd1-42d6-473d-9568-ad6ba68ca22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 04 01:22:47.525216 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:47.520215 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded" Nov 04 01:22:47.525216 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:47.519059 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Nov 04 01:22:47.525216 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:47.520290 7059 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 04 01:22:47.525216 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:47.521178 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" podUID="cce88e25-4d81-4b48-a229-b72e308c5f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 04 01:22:47.633952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:47.619765 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Nov 04 01:22:47.753422 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:47.753325 7059 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 04 01:22:47.753669 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:47.753633 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" podUID="eedbad81-5540-45d7-adbb-54afd75aa6e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 04 01:22:47.754461 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:47.754438 7059 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:22:47.754637 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:47.754618 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" podUID="b42e3b72-1c9f-4a14-88c8-755b3faceb5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 04 01:22:47.806949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:47.806029 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 04 01:22:47.806949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:47.806240 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 04 01:22:47.893071 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:47.893018 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:48.010997 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:47.976909 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:22:48.010997 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:48.002397 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:22:48.010997 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:48.002449 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:22:48.039950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:48.034908 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 04 01:22:48.139614 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:48.044014 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 04 01:22:48.139614 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:48.048245 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:48.139614 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:48.095889 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:48.174008 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:48.172322 7059 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:22:48.174008 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:48.172384 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" podUID="19a17fd1-42d6-473d-9568-ad6ba68ca22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 04 01:22:48.174421 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:48.174396 7059 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 04 01:22:48.174567 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:48.174545 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" podUID="cce88e25-4d81-4b48-a229-b72e308c5f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 04 01:22:48.225282 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:48.208336 7059 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:22:48.225282 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:48.208394 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" podUID="b42e3b72-1c9f-4a14-88c8-755b3faceb5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 04 01:22:48.393072 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:48.392111 7059 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 04 01:22:48.393072 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:48.392175 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" podUID="eedbad81-5540-45d7-adbb-54afd75aa6e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 04 01:22:48.648124 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:48.648016 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:48.705064 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:48.659160 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded" Nov 04 01:22:48.795951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:48.665546 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Nov 04 01:22:49.040155 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:49.040094 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:49.063946 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:49.045043 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 04 01:22:49.089872 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:49.058096 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 04 01:22:49.091508 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:49.091484 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:22:49.144989 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:49.143036 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:49.144989 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:49.143107 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:49.287556 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:49.287512 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 04 01:22:49.771378 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:49.390175 7059 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:22:49.776169 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:49.776127 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" podUID="19a17fd1-42d6-473d-9568-ad6ba68ca22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 04 01:22:49.793831 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:49.400354 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:22:49.794450 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:49.457159 7059 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 04 01:22:49.794645 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:49.794608 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" podUID="eedbad81-5540-45d7-adbb-54afd75aa6e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 04 01:22:49.794780 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:49.528035 7059 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 04 01:22:49.794890 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:49.794871 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" podUID="cce88e25-4d81-4b48-a229-b72e308c5f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 04 01:22:49.795015 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:49.487914 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:49.795127 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:49.717121 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Nov 04 01:22:49.795498 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:49.760802 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:49.816077 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:49.816031 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:49.824955 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:49.820051 7059 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:22:49.826391 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:49.826344 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" podUID="b42e3b72-1c9f-4a14-88c8-755b3faceb5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 04 01:22:50.096020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:50.071566 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 04 01:22:50.131341 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:50.093208 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:50.133508 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:50.094517 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 04 01:22:50.161181 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:50.161142 7059 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 04 01:22:50.161463 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:50.161433 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" podUID="cce88e25-4d81-4b48-a229-b72e308c5f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 04 01:22:50.178598 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:50.178564 7059 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:22:50.178964 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:50.178869 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" podUID="19a17fd1-42d6-473d-9568-ad6ba68ca22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 04 01:22:50.179363 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:50.179181 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:50.179666 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:50.179226 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 04 01:22:50.244950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:50.244657 7059 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:22:50.244950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:50.244740 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" podUID="b42e3b72-1c9f-4a14-88c8-755b3faceb5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 04 01:22:50.248948 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:22:50.248211 7059 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.06s" Nov 04 01:22:50.306747 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:50.306699 7059 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 04 01:22:50.321950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:50.318626 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" podUID="eedbad81-5540-45d7-adbb-54afd75aa6e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 04 01:22:50.448358 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:50.448311 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:50.645067 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:50.645019 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:50.808124 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:50.806859 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:50.808124 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:50.807998 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Nov 04 01:22:50.808660 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:50.808302 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded" Nov 04 01:22:50.830911 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:50.824984 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:22:50.851448 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:50.848655 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Nov 04 01:22:50.874481 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:50.874038 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:51.238950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:51.223426 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:51.355950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:51.355626 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:51.362239 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:51.360005 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 04 01:22:51.385957 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:51.362630 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 04 01:22:51.411061 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:51.403083 7059 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 04 01:22:51.411061 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:51.403139 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" podUID="eedbad81-5540-45d7-adbb-54afd75aa6e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 04 01:22:51.437956 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:51.437104 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 04 01:22:51.460956 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:51.458769 7059 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:22:51.460956 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:51.458825 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" podUID="b42e3b72-1c9f-4a14-88c8-755b3faceb5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 04 01:22:51.483305 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:51.471164 7059 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 04 01:22:51.483305 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:51.471221 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" podUID="cce88e25-4d81-4b48-a229-b72e308c5f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 04 01:22:51.522446 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:51.514495 7059 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:22:51.522446 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:51.514555 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" podUID="19a17fd1-42d6-473d-9568-ad6ba68ca22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 04 01:22:51.641497 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:51.641393 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4r4nr" podUID="ba1d2c7e-87da-4142-82f3-9a378b73f3b3" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.30:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:51.782948 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:51.782846 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:22:51.783420 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:51.783398 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:22:51.808163 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:51.808114 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:51.809669 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:51.809636 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded" Nov 04 01:22:51.810908 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:51.809736 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:51.847334 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:51.847286 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:22:51.872678 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:51.872639 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:51.883947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:51.876375 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:22:51.924173 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:51.922690 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:22:51.924173 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:51.922784 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:22:51.932979 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:51.932563 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:22:52.102772 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:52.102688 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:22:52.112365 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:52.112339 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:22:52.119635 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:52.119614 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:22:52.145431 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:52.145033 7059 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 04 01:22:52.145431 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:52.145090 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" podUID="cce88e25-4d81-4b48-a229-b72e308c5f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 04 01:22:52.172110 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:52.172068 7059 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:22:52.172383 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:52.172351 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" podUID="19a17fd1-42d6-473d-9568-ad6ba68ca22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 04 01:22:52.198848 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:52.198443 7059 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:22:52.198848 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:52.198501 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" podUID="b42e3b72-1c9f-4a14-88c8-755b3faceb5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 04 01:22:52.270112 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:52.270077 7059 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Nov 04 01:22:52.270369 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:52.270339 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" podUID="eedbad81-5540-45d7-adbb-54afd75aa6e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Nov 04 01:22:53.182982 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:53.182949 7059 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:22:53.183624 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:53.183584 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" podUID="19a17fd1-42d6-473d-9568-ad6ba68ca22a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 04 01:22:53.318243 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:53.318202 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:22:53.320761 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:53.320732 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:22:53.324613 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:53.324586 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:22:53.901309 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:53.901275 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:22:53.905704 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:53.903232 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:22:53.906687 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:53.906648 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:22:53.916315 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:53.916224 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:22:53.916681 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:53.916664 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:22:53.917823 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:53.917804 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:22:54.174450 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:54.174419 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:22:54.174821 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:54.174799 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:22:54.175368 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:54.175353 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:22:58.759952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:58.756045 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:58.777955 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:58.777893 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:58.806167 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:58.806121 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:58.844951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:58.841841 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:59.050010 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:59.049037 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:59.094947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:59.094196 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 04 01:22:59.233950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:59.224365 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:59.233950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:59.231284 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 04 01:22:59.379289 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:59.379189 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:59.408469 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:59.403147 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:59.770951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:59.761052 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:59.770951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:59.761860 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:22:59.790949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:59.787846 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:59.790949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:59.788459 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:22:59.820129 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:59.818724 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:59.820129 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:59.819097 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:22:59.838220 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:59.837751 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:59.843665 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:59.843619 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:59.844114 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:22:59.844096 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:23:00.064017 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:00.049724 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:00.088952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:00.065402 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:23:00.116083 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:00.116032 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:00.120686 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:00.117536 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:23:00.233976 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:00.229119 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:00.237953 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:00.234289 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:23:00.254983 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:00.234279 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 04 01:23:00.279512 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:00.277358 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:23:00.383080 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:00.367957 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:00.432957 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:00.410086 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:00.432957 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:00.411447 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:23:00.658998 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:00.634630 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:00.838954 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:00.778885 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Nov 04 01:23:00.838954 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:00.796495 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:00.838954 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:00.830400 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded" Nov 04 01:23:00.874960 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:00.843055 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:00.874960 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:00.869517 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:00.917060 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:00.870293 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:23:01.069049 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:01.069002 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 04 01:23:01.120291 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:01.120231 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:01.145970 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:01.121436 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4r4nr" podUID="ba1d2c7e-87da-4142-82f3-9a378b73f3b3" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.30:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:01.269009 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:01.246437 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:01.291055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:01.291007 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:01.553007 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:01.479002 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 04 01:23:01.736091 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:01.736045 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4r4nr" podUID="ba1d2c7e-87da-4142-82f3-9a378b73f3b3" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.30:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:01.804132 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:01.804028 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:01.861951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:01.858297 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:02.028103 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:01.908081 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:02.141011 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:01.954576 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Nov 04 01:23:02.211314 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:02.007279 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:02.211817 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:02.084391 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 04 01:23:02.219952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:02.124352 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 04 01:23:02.285086 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:02.285040 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:02.419088 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:02.386429 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:02.524952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:02.485115 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 04 01:23:02.815085 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:02.815035 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:03.033721 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:03.032552 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Nov 04 01:23:03.069128 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:03.051114 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:03.213184 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:03.213132 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 04 01:23:03.213782 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:03.213746 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Nov 04 01:23:03.214243 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:03.214218 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:23:03.254952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:03.246560 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 04 01:23:03.286072 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:03.286024 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:03.443953 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:03.437523 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 04 01:23:03.497858 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:03.497810 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 04 01:23:03.841996 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:03.841850 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:04.036053 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:04.036005 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:04.058425 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:04.055006 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:04.094643 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:04.093648 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:04.185136 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:04.167961 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:04.224273 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:04.224224 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:04.224875 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:04.224844 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 04 01:23:04.229330 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:04.229295 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:04.269937 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:04.269868 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 04 01:23:04.302367 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:04.302323 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 04 01:23:04.478027 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:04.440829 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:04.514955 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:04.501314 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 04 01:23:04.875638 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:04.875531 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:05.044144 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:05.044095 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:05.055852 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:05.055797 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:05.228053 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:05.228007 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:05.229689 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:05.229648 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:05.233998 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:05.230166 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 04 01:23:05.280635 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:05.276345 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:05.332255 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:05.322085 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 04 01:23:05.467384 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:05.453176 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:05.572140 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:05.572043 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:05.896698 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:05.896600 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:05.988997 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:05.988030 7059 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:05.988997 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:05.988100 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" podUID="7e59c4bd-a00f-41e1-8d1c-610b9d791374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:06.047551 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:06.047270 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Nov 04 01:23:06.063093 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:06.062250 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:06.107962 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:06.100375 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": context deadline exceeded" Nov 04 01:23:06.257039 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:06.247602 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:06.257039 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:06.248004 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Nov 04 01:23:06.295236 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:06.295082 7059 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:06.295236 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:06.295155 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" podUID="f0f8924f-e452-4b2c-8b41-f970764ff963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:06.348855 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:06.297826 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:23:06.348855 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:06.322864 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 04 01:23:06.391157 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:06.369260 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:06.512193 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:06.485176 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:06.774294 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:06.590357 7059 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:06.774294 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:06.590421 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" podUID="b59977fd-6792-40e9-9070-1f04beb3f0bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:06.774294 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:06.593497 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 04 01:23:06.774294 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:06.615979 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:06.774294 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:06.661080 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:06.774294 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:06.739055 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": context deadline exceeded" Nov 04 01:23:06.774294 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:06.773181 7059 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:06.774294 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:06.773257 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" podUID="49cff42c-6b14-409e-b5a1-9eb70c788da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:06.935198 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:06.932899 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:07.023537 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:07.023500 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:23:07.033104 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:23:07.033021 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:23:07.127997 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:07.038697 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:23:07.226574 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:07.141017 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:07.261239 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:07.261199 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Nov 04 01:23:07.368354 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:07.303596 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 04 01:23:07.369225 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:07.321037 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:07.369546 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:07.369510 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:07.372908 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:07.372333 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:07.445840 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:07.445799 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:23:07.446119 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:07.446094 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:23:07.496496 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:07.496469 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:23:07.498960 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:07.497586 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:07.521871 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:07.521829 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:23:07.522222 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:07.522204 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:23:07.523350 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:07.522648 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:23:07.523350 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:07.523083 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:23:07.524226 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:07.524200 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:23:29.093950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:29.073038 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:29.093950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:29.074026 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:29.105955 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:29.098039 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded" Nov 04 01:23:29.157269 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:29.157226 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:29.305150 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:29.285248 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:36.483910 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:23:36.483870 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:23:50.640454 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:50.640414 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct"] Nov 04 01:23:50.646406 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:50.646385 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct" Nov 04 01:23:50.668182 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:50.668159 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 04 01:23:50.668343 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:50.668296 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 04 01:23:50.668412 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:50.668298 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 04 01:23:50.668746 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:50.668435 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-kwn4w\"" Nov 04 01:23:50.673554 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:50.673533 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct"] Nov 04 01:23:50.728462 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:50.728428 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4snct\" (UID: \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct" Nov 04 01:23:50.728628 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:50.728471 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4snct\" (UID: \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct" Nov 04 01:23:50.728628 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:50.728598 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4snct\" (UID: \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct" Nov 04 01:23:50.728765 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:50.728673 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4snct\" (UID: \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct" Nov 04 01:23:50.728765 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:50.728699 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjd42\" (UniqueName: \"kubernetes.io/projected/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-kube-api-access-gjd42\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4snct\" (UID: \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct" Nov 04 01:23:50.728878 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:50.728849 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4snct\" (UID: \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct" Nov 04 01:23:50.728963 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:50.728910 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4snct\" (UID: \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct" Nov 04 01:23:50.829864 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:50.829832 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4snct\" (UID: \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct" Nov 04 01:23:50.830024 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:50.829889 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4snct\" (UID: \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct" Nov 04 01:23:50.830024 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:50.830004 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4snct\" (UID: \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct" Nov 04 01:23:50.830193 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:50.830106 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4snct\" (UID: \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct" Nov 04 01:23:50.830193 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:50.830137 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjd42\" (UniqueName: \"kubernetes.io/projected/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-kube-api-access-gjd42\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4snct\" (UID: \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct" Nov 04 01:23:50.830312 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:50.830207 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4snct\" (UID: \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct" Nov 04 01:23:50.830312 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:50.830251 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4snct\" (UID: \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct" Nov 04 01:23:50.831009 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:50.830983 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4snct\" (UID: \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct" Nov 04 01:23:50.832820 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:50.832799 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4snct\" (UID: \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct" Nov 04 01:23:50.833030 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:50.833010 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4snct\" (UID: \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct" Nov 04 01:23:50.833098 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:50.833066 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4snct\" (UID: \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct" Nov 04 01:23:50.833145 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:50.833125 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4snct\" (UID: \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct" Nov 04 01:23:50.840886 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:50.840860 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4snct\" (UID: \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct" Nov 04 01:23:50.841904 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:50.841888 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjd42\" (UniqueName: \"kubernetes.io/projected/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-kube-api-access-gjd42\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4snct\" (UID: \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct" Nov 04 01:23:50.960762 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:50.960729 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct" Nov 04 01:23:51.249070 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:51.249042 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct"] Nov 04 01:23:51.249208 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:23:51.249137 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab0633d_d9de_42ab_a0f6_d64bfe8ecafd.slice/crio-6ac55464ab0e26f7b19d5473faefe83670eeff8b5008db524d363630f64e07ca WatchSource:0}: Error finding container 6ac55464ab0e26f7b19d5473faefe83670eeff8b5008db524d363630f64e07ca: Status 404 returned error can't find the container with id 6ac55464ab0e26f7b19d5473faefe83670eeff8b5008db524d363630f64e07ca Nov 04 01:23:51.251302 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:51.251284 7059 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:23:51.931131 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:51.931095 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct" event={"ID":"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd","Type":"ContainerStarted","Data":"6ac55464ab0e26f7b19d5473faefe83670eeff8b5008db524d363630f64e07ca"} Nov 04 01:23:55.063121 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:55.063088 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:23:55.063411 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:55.063150 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:23:55.973534 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:55.973485 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct" event={"ID":"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd","Type":"ContainerStarted","Data":"53099f172a42ae74621e1c7a36702e25cf99cae32b19eb3a0d4c502a62bed227"} Nov 04 01:23:55.973799 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:55.973551 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct" Nov 04 01:23:56.008143 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:56.008082 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct" podStartSLOduration=2.197886796 podStartE2EDuration="6.008066669s" totalImagesPullingTime="3.810179873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:23:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:23:56.004155388 +0000 UTC m=+1520.946004292" watchObservedRunningTime="2026-11-04 01:23:56.008066669 +0000 UTC m=+1520.949915571" Nov 04 01:23:56.989351 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:23:56.989316 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct" Nov 04 01:24:06.517990 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:24:06.517949 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:24:09.090239 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:09.090188 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:09.234518 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:09.234462 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:09.410899 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:09.410789 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:09.450963 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:09.447729 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:09.494946 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:09.492106 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:09.783946 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:09.783632 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Nov 04 01:24:09.828950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:09.811013 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:10.055462 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:10.041057 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:10.055462 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:10.046045 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:10.111965 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:10.095031 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 04 01:24:10.111965 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:10.095692 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:24:10.142160 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:10.142084 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 04 01:24:10.353894 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:10.353023 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": context deadline exceeded" Nov 04 01:24:10.353894 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:10.353138 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:24:10.414966 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:10.359861 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:24:10.414966 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:10.409233 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:10.514626 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:10.459279 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:10.514626 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:10.459768 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:24:10.583973 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:10.541724 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 04 01:24:10.583973 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:10.542237 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:24:10.643460 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:10.643352 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:10.850949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:10.839351 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Nov 04 01:24:10.895078 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:10.840736 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:10.895078 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:10.858580 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:24:10.895078 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:10.858642 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:24:11.094036 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:11.076528 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:11.118016 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:11.104663 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:24:11.121429 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:11.121390 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:11.126418 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:11.125342 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:11.126943 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:11.126777 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:24:11.315971 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:11.312508 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 04 01:24:11.379957 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:11.359166 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:24:11.458610 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:11.458556 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 04 01:24:11.540488 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:11.465341 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:11.552071 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:11.552008 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:11.586802 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:11.586710 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:24:11.639471 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:11.638023 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:24:11.895700 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:11.895603 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 04 01:24:11.986608 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:11.895945 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:12.124442 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:12.124397 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:12.132952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:12.128892 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:12.133544 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:12.133513 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded" Nov 04 01:24:12.312888 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:12.312842 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:12.395958 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:12.386148 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:12.691961 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:12.683420 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:12.792772 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:12.775812 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 04 01:24:12.811256 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:12.811205 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:13.157008 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:13.156092 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:13.203952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:13.174585 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:13.203952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:13.202403 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 04 01:24:13.232357 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:13.232314 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Nov 04 01:24:13.233561 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:13.233520 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 04 01:24:13.345111 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:13.345066 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:13.610239 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:13.610193 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:13.817531 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:13.756211 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 04 01:24:13.870715 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:13.870601 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:13.871084 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:13.870740 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:14.103169 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:14.081964 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:14.160189 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:14.160078 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:14.209317 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:14.209268 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:14.266816 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:14.266757 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:14.300034 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:14.269393 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 04 01:24:14.300595 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:14.269520 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:14.358956 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:14.356019 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:14.437069 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:14.437019 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:14.820036 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:14.819040 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:24:14.868945 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:14.864021 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:14.872470 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:14.872423 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:14.879959 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:14.872680 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 04 01:24:15.169037 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:15.165211 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:15.241090 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:15.241039 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:15.306952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:15.274012 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded" Nov 04 01:24:15.309675 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:15.309634 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:15.310264 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:15.310236 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 04 01:24:15.374433 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:15.374367 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:15.820238 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:15.820178 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:15.874144 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:15.874092 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 04 01:24:15.874968 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:15.874941 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 04 01:24:15.884735 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:15.884702 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:16.140840 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:16.135315 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:16.184966 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:16.176010 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:16.247547 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:16.247506 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:16.275620 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:16.275575 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:16.315148 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:16.315099 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:16.315774 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:16.315738 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:16.379945 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:16.378800 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:24:16.394062 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:16.380369 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:16.575950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:16.570347 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:24:16.614824 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:16.611371 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:16.637066 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:16.636906 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:24:16.640298 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:16.640067 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:16.786154 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:16.786119 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:24:16.791453 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:16.791253 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:24:16.793661 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:16.793641 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:24:16.830563 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:16.830531 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:24:16.831604 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:16.831577 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:24:16.832377 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:16.832353 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:24:16.832558 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:16.832421 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:24:36.483237 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:24:36.483191 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:24:42.271043 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:42.271005 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn"] Nov 04 01:24:42.276132 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:42.276107 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn" Nov 04 01:24:42.295621 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:42.295489 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-7b10de73-serving-cert\"" Nov 04 01:24:42.299719 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:42.299690 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn"] Nov 04 01:24:42.371974 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:42.371901 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8099eb0-eb29-4e6f-8715-d714104a0ff1-openshift-service-ca\") pod \"maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn\" (UID: \"f8099eb0-eb29-4e6f-8715-d714104a0ff1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn" Nov 04 01:24:42.372227 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:42.372200 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8099eb0-eb29-4e6f-8715-d714104a0ff1-metrics-certs\") pod \"maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn\" (UID: \"f8099eb0-eb29-4e6f-8715-d714104a0ff1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn" Nov 04 01:24:42.372343 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:42.372302 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw5wk\" (UniqueName: \"kubernetes.io/projected/f8099eb0-eb29-4e6f-8715-d714104a0ff1-kube-api-access-nw5wk\") pod \"maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn\" (UID: \"f8099eb0-eb29-4e6f-8715-d714104a0ff1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn" Nov 04 01:24:42.372402 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:42.372387 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8099eb0-eb29-4e6f-8715-d714104a0ff1-maas-api-tls\") pod \"maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn\" (UID: \"f8099eb0-eb29-4e6f-8715-d714104a0ff1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn" Nov 04 01:24:42.474227 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:42.474194 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8099eb0-eb29-4e6f-8715-d714104a0ff1-metrics-certs\") pod \"maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn\" (UID: \"f8099eb0-eb29-4e6f-8715-d714104a0ff1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn" Nov 04 01:24:42.474436 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:42.474306 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw5wk\" (UniqueName: \"kubernetes.io/projected/f8099eb0-eb29-4e6f-8715-d714104a0ff1-kube-api-access-nw5wk\") pod \"maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn\" (UID: \"f8099eb0-eb29-4e6f-8715-d714104a0ff1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn" Nov 04 01:24:42.474436 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:42.474360 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8099eb0-eb29-4e6f-8715-d714104a0ff1-maas-api-tls\") pod \"maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn\" (UID: \"f8099eb0-eb29-4e6f-8715-d714104a0ff1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn" Nov 04 01:24:42.474562 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:42.474544 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8099eb0-eb29-4e6f-8715-d714104a0ff1-openshift-service-ca\") pod \"maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn\" (UID: \"f8099eb0-eb29-4e6f-8715-d714104a0ff1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn" Nov 04 01:24:42.475226 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:42.475203 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8099eb0-eb29-4e6f-8715-d714104a0ff1-openshift-service-ca\") pod \"maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn\" (UID: \"f8099eb0-eb29-4e6f-8715-d714104a0ff1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn" Nov 04 01:24:42.476938 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:42.476896 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8099eb0-eb29-4e6f-8715-d714104a0ff1-metrics-certs\") pod \"maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn\" (UID: \"f8099eb0-eb29-4e6f-8715-d714104a0ff1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn" Nov 04 01:24:42.477083 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:42.477065 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8099eb0-eb29-4e6f-8715-d714104a0ff1-maas-api-tls\") pod \"maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn\" (UID: \"f8099eb0-eb29-4e6f-8715-d714104a0ff1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn" Nov 04 01:24:42.492520 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:42.492494 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw5wk\" (UniqueName: \"kubernetes.io/projected/f8099eb0-eb29-4e6f-8715-d714104a0ff1-kube-api-access-nw5wk\") pod \"maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn\" (UID: \"f8099eb0-eb29-4e6f-8715-d714104a0ff1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn" Nov 04 01:24:42.603174 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:42.603100 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn" Nov 04 01:24:42.905944 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:42.905891 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn"] Nov 04 01:24:42.907612 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:24:42.907572 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8099eb0_eb29_4e6f_8715_d714104a0ff1.slice/crio-1b1eaab20e5725d6c0b9f9bfc181ed357dc328b437eb1e8ef7957a19835296db WatchSource:0}: Error finding container 1b1eaab20e5725d6c0b9f9bfc181ed357dc328b437eb1e8ef7957a19835296db: Status 404 returned error can't find the container with id 1b1eaab20e5725d6c0b9f9bfc181ed357dc328b437eb1e8ef7957a19835296db Nov 04 01:24:43.430665 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:43.430624 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn" event={"ID":"f8099eb0-eb29-4e6f-8715-d714104a0ff1","Type":"ContainerStarted","Data":"1b1eaab20e5725d6c0b9f9bfc181ed357dc328b437eb1e8ef7957a19835296db"} Nov 04 01:24:44.542950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:44.542313 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn" event={"ID":"f8099eb0-eb29-4e6f-8715-d714104a0ff1","Type":"ContainerStarted","Data":"eb0bcf0a7c6feb8fa63af6c3d3ac6ea28568fe4f4d3e707d4ac171638a3c16ae"} Nov 04 01:24:44.542950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:44.542650 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn" Nov 04 01:24:44.611962 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:44.598849 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn" podStartSLOduration=2.102783379 podStartE2EDuration="2.598825786s" totalImagesPullingTime="496.042407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:24:44.572788975 +0000 UTC m=+1569.514637891" watchObservedRunningTime="2026-11-04 01:24:44.598825786 +0000 UTC m=+1569.540674690" Nov 04 01:24:44.778072 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:44.778020 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:44.822118 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:44.822015 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:44.873077 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:44.873027 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:44.873444 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:44.873415 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Nov 04 01:24:45.005003 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:45.000314 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:45.129014 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:45.127011 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 04 01:24:45.163912 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:45.163870 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded" Nov 04 01:24:45.334319 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:45.334275 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:45.365065 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:45.365019 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:45.365687 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:45.365656 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 04 01:24:45.914955 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:45.877224 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:45.928950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:45.917058 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 04 01:24:45.983962 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:45.922621 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:46.154454 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:45.923393 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:24:46.154785 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:46.154761 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:24:46.154996 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:46.154980 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:24:46.156293 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:45.923842 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:46.158584 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:46.021293 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:46.275402 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:46.174421 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:24:46.278950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:46.275749 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:24:46.302957 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:46.111311 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:46.360235 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:46.174494 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:46.380520 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:46.209562 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:46.419716 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:46.419671 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:46.420779 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:46.420747 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:46.702950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:46.681590 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": context deadline exceeded" Nov 04 01:24:46.718429 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:46.706063 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:46.722018 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:46.718025 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:46.741009 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:46.718139 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:24:46.741348 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:46.741326 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:24:46.741533 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:46.741520 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:24:46.741960 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:46.741917 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:24:46.784429 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:46.784384 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": context deadline exceeded" Nov 04 01:24:47.033061 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:46.887600 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:24:47.033061 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:46.978061 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4r4nr" podUID="ba1d2c7e-87da-4142-82f3-9a378b73f3b3" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.30:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:47.033061 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:46.979529 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:47.033061 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:47.012191 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:47.178989 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:47.175223 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 04 01:24:47.178989 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:47.175416 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 04 01:24:47.597039 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:47.563096 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:47.784983 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:47.777106 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:24:48.261951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:47.807058 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 04 01:24:48.280217 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:48.011055 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Nov 04 01:24:48.280681 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:48.149050 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": context deadline exceeded" Nov 04 01:24:48.281035 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:48.243132 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 04 01:24:48.294987 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:48.210089 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded" Nov 04 01:24:49.518951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:49.459999 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 04 01:24:49.518951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:49.460405 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Nov 04 01:24:49.518951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:49.460675 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 04 01:24:49.518951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:49.460953 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 04 01:24:49.518951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:49.469722 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:49.518951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:49.481202 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:49.518951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:49.498433 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:49.518951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:49.498514 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 04 01:24:49.518951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:49.498822 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Nov 04 01:24:49.518951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:49.499101 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 04 01:24:49.518951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:49.499349 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded" Nov 04 01:24:50.427540 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:50.427500 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": context deadline exceeded" Nov 04 01:24:50.463955 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:50.463041 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:50.463955 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:50.463105 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:50.464236 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:50.463979 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:50.465178 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:50.464318 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:50.476952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:50.470354 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:50.492185 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:50.487032 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:50.505123 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:50.505082 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 04 01:24:50.505596 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:50.505572 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Nov 04 01:24:50.506049 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:50.506022 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 04 01:24:50.506466 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:50.506448 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded" Nov 04 01:24:50.644314 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:50.643235 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:50.724126 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:50.724032 7059 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:50.724391 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:50.724364 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" podUID="0085890d-7ef5-4947-975c-440894141023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:50.724955 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:50.724934 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn" Nov 04 01:24:51.290055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:51.290012 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4r4nr" podUID="ba1d2c7e-87da-4142-82f3-9a378b73f3b3" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.30:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:51.471319 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:51.471273 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:51.472074 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:51.472042 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 04 01:24:51.493288 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:51.493244 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:51.512266 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:51.512225 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:24:51.529943 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:51.517000 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 04 01:24:51.539959 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:51.520899 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:51.564007 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:51.521138 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 04 01:24:51.568956 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:51.521370 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:51.568956 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:51.521579 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 04 01:24:51.568956 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:51.521820 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:52.161180 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:52.160436 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:24:52.181940 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:52.181783 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:24:52.190715 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:52.190692 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:24:52.191958 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:52.191911 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:24:52.195227 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:52.195202 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:24:52.200881 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:52.200810 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:24:52.201980 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:52.201955 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:24:52.203179 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:52.203144 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:24:52.211979 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:52.205353 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:24:52.211979 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:24:52.206056 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:25:06.482905 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:25:06.482867 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:25:15.327340 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:15.327272 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:15.355207 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:15.349081 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded" Nov 04 01:25:15.399991 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:15.399908 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:15.501780 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:15.478515 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 04 01:25:15.510222 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:15.510098 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:15.550621 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:15.550048 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:15.638012 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:15.637102 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:16.135944 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:16.104826 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:16.241562 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:16.241511 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:16.442845 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:16.442804 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:25:16.471326 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:16.471292 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:25:16.570949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:16.570375 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 04 01:25:16.677074 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:16.645752 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 04 01:25:16.677074 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:16.647559 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:25:16.931009 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:16.892269 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:16.941887 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:16.941846 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Nov 04 01:25:16.954988 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:16.937955 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:25:16.958948 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:16.955368 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:25:17.003159 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:16.995864 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": context deadline exceeded" Nov 04 01:25:17.232004 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:17.213339 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:17.271047 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:17.242391 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": context deadline exceeded" Nov 04 01:25:17.271418 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:17.243849 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 04 01:25:17.271581 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:17.270592 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:17.352297 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:17.352247 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded" Nov 04 01:25:17.438950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:17.431606 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:17.467676 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:17.467632 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:25:17.780965 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:17.650170 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:17.797079 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:17.673026 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:17.951085 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:17.951038 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:17.968951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:17.956377 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:18.010948 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:17.985999 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:25:18.010948 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:17.986226 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:25:18.010948 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:17.986278 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:25:18.242439 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:18.242388 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 04 01:25:18.271154 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:18.271121 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:25:18.292316 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:18.290095 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:18.426950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:18.426545 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded" Nov 04 01:25:18.427150 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:18.426967 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:25:18.456951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:18.456384 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:18.505999 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:18.502465 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:25:18.517085 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:18.517025 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:18.586116 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:18.586060 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:18.690057 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:18.690018 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 04 01:25:18.784997 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:18.784202 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:18.812952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:18.798027 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:19.020032 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:19.014053 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 04 01:25:19.053331 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:19.044373 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:19.381746 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:19.381191 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 04 01:25:19.421950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:19.421854 7059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-ddd7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:25:19.421950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:19.421939 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" podUID="26881d61-360e-43bb-83f8-3122e0026e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:19.437946 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:19.432000 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:19.437946 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:19.432368 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded" Nov 04 01:25:19.457415 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:19.457374 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:19.537435 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:19.524356 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:25:19.590192 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:19.590149 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:19.694113 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:19.691593 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:19.816821 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:19.745637 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:25:19.827111 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:19.802148 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 04 01:25:19.827888 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:19.827866 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:25:20.024320 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:20.023324 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:20.051062 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:20.048178 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:25:20.051062 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:20.049517 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:20.386103 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:20.386010 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:20.386557 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:20.386524 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 04 01:25:20.461954 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:20.429625 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" Nov 04 01:25:20.461954 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:20.433512 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:25:20.596975 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:20.569853 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 04 01:25:20.596975 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:20.571797 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:20.696054 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:20.685796 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:25:20.699080 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:20.697398 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:20.741957 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:20.737975 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": context deadline exceeded" Nov 04 01:25:20.959154 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:20.959061 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Nov 04 01:25:21.021991 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:21.021952 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:25:21.051773 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:21.051730 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Nov 04 01:25:21.116318 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:21.116286 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:25:21.131453 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:21.131419 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:25:21.154583 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:21.154553 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:25:23.677634 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:23.677594 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn"] Nov 04 01:25:23.678159 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:23.677835 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn" podUID="f8099eb0-eb29-4e6f-8715-d714104a0ff1" containerName="maas-api" containerID="cri-o://eb0bcf0a7c6feb8fa63af6c3d3ac6ea28568fe4f4d3e707d4ac171638a3c16ae" gracePeriod=30 Nov 04 01:25:24.096397 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:24.096372 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn" Nov 04 01:25:24.160388 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:24.160349 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8099eb0-eb29-4e6f-8715-d714104a0ff1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8099eb0-eb29-4e6f-8715-d714104a0ff1-metrics-certs\") pod \"f8099eb0-eb29-4e6f-8715-d714104a0ff1\" (UID: \"f8099eb0-eb29-4e6f-8715-d714104a0ff1\") " Nov 04 01:25:24.160565 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:24.160498 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8099eb0-eb29-4e6f-8715-d714104a0ff1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8099eb0-eb29-4e6f-8715-d714104a0ff1-openshift-service-ca\") pod \"f8099eb0-eb29-4e6f-8715-d714104a0ff1\" (UID: \"f8099eb0-eb29-4e6f-8715-d714104a0ff1\") " Nov 04 01:25:24.160622 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:24.160582 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8099eb0-eb29-4e6f-8715-d714104a0ff1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8099eb0-eb29-4e6f-8715-d714104a0ff1-maas-api-tls\") pod \"f8099eb0-eb29-4e6f-8715-d714104a0ff1\" (UID: \"f8099eb0-eb29-4e6f-8715-d714104a0ff1\") " Nov 04 01:25:24.160676 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:24.160661 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8099eb0-eb29-4e6f-8715-d714104a0ff1-kube-api-access-nw5wk\" (UniqueName: \"kubernetes.io/projected/f8099eb0-eb29-4e6f-8715-d714104a0ff1-kube-api-access-nw5wk\") pod \"f8099eb0-eb29-4e6f-8715-d714104a0ff1\" (UID: \"f8099eb0-eb29-4e6f-8715-d714104a0ff1\") " Nov 04 01:25:24.162043 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:24.161960 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8099eb0-eb29-4e6f-8715-d714104a0ff1-openshift-service-ca" pod "f8099eb0-eb29-4e6f-8715-d714104a0ff1" (UID: "f8099eb0-eb29-4e6f-8715-d714104a0ff1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:25:24.171001 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:24.170385 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8099eb0-eb29-4e6f-8715-d714104a0ff1-kube-api-access-nw5wk" pod "f8099eb0-eb29-4e6f-8715-d714104a0ff1" (UID: "f8099eb0-eb29-4e6f-8715-d714104a0ff1"). InnerVolumeSpecName "kube-api-access-nw5wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:25:24.171001 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:24.170798 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8099eb0-eb29-4e6f-8715-d714104a0ff1-maas-api-tls" pod "f8099eb0-eb29-4e6f-8715-d714104a0ff1" (UID: "f8099eb0-eb29-4e6f-8715-d714104a0ff1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:25:24.171240 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:24.171136 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8099eb0-eb29-4e6f-8715-d714104a0ff1-metrics-certs" pod "f8099eb0-eb29-4e6f-8715-d714104a0ff1" (UID: "f8099eb0-eb29-4e6f-8715-d714104a0ff1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:25:24.262808 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:24.262726 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw5wk\" (UniqueName: \"kubernetes.io/projected/f8099eb0-eb29-4e6f-8715-d714104a0ff1-kube-api-access-nw5wk\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:24.262808 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:24.262756 7059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8099eb0-eb29-4e6f-8715-d714104a0ff1-metrics-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:24.262808 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:24.262767 7059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8099eb0-eb29-4e6f-8715-d714104a0ff1-openshift-service-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:24.262808 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:24.262776 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8099eb0-eb29-4e6f-8715-d714104a0ff1-maas-api-tls\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:24.776741 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:24.776705 7059 generic.go:356] "Generic (PLEG): container finished" podID="f8099eb0-eb29-4e6f-8715-d714104a0ff1" containerID="eb0bcf0a7c6feb8fa63af6c3d3ac6ea28568fe4f4d3e707d4ac171638a3c16ae" exitCode=0 Nov 04 01:25:24.777187 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:24.776757 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn" Nov 04 01:25:24.777187 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:24.776791 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn" event={"ID":"f8099eb0-eb29-4e6f-8715-d714104a0ff1","Type":"ContainerDied","Data":"eb0bcf0a7c6feb8fa63af6c3d3ac6ea28568fe4f4d3e707d4ac171638a3c16ae"} Nov 04 01:25:24.777187 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:24.776851 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn" event={"ID":"f8099eb0-eb29-4e6f-8715-d714104a0ff1","Type":"ContainerDied","Data":"1b1eaab20e5725d6c0b9f9bfc181ed357dc328b437eb1e8ef7957a19835296db"} Nov 04 01:25:24.777187 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:24.776878 7059 scope.go:122] "RemoveContainer" containerID="eb0bcf0a7c6feb8fa63af6c3d3ac6ea28568fe4f4d3e707d4ac171638a3c16ae" Nov 04 01:25:24.789335 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:24.789309 7059 scope.go:122] "RemoveContainer" containerID="eb0bcf0a7c6feb8fa63af6c3d3ac6ea28568fe4f4d3e707d4ac171638a3c16ae" Nov 04 01:25:24.789644 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:25:24.789625 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb0bcf0a7c6feb8fa63af6c3d3ac6ea28568fe4f4d3e707d4ac171638a3c16ae\": container with ID starting with eb0bcf0a7c6feb8fa63af6c3d3ac6ea28568fe4f4d3e707d4ac171638a3c16ae not found: ID does not exist" containerID="eb0bcf0a7c6feb8fa63af6c3d3ac6ea28568fe4f4d3e707d4ac171638a3c16ae" Nov 04 01:25:24.789712 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:24.789651 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb0bcf0a7c6feb8fa63af6c3d3ac6ea28568fe4f4d3e707d4ac171638a3c16ae"} err="failed to get container status \"eb0bcf0a7c6feb8fa63af6c3d3ac6ea28568fe4f4d3e707d4ac171638a3c16ae\": rpc error: code = NotFound desc = could not find container \"eb0bcf0a7c6feb8fa63af6c3d3ac6ea28568fe4f4d3e707d4ac171638a3c16ae\": container with ID starting with eb0bcf0a7c6feb8fa63af6c3d3ac6ea28568fe4f4d3e707d4ac171638a3c16ae not found: ID does not exist" Nov 04 01:25:24.825441 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:24.825412 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn"] Nov 04 01:25:24.840863 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:24.840826 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-7b10de73-846fcd5549-fmgwn"] Nov 04 01:25:25.196338 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:25.196298 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8099eb0-eb29-4e6f-8715-d714104a0ff1" path="/var/lib/kubelet/pods/f8099eb0-eb29-4e6f-8715-d714104a0ff1/volumes" Nov 04 01:25:35.242763 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.242725 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2"] Nov 04 01:25:35.243656 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.243620 7059 cpu_manager.go:414] "containerMap: removing container" podUID="f8099eb0-eb29-4e6f-8715-d714104a0ff1" containerName="maas-api" Nov 04 01:25:35.243715 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.243658 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8099eb0-eb29-4e6f-8715-d714104a0ff1" containerName="maas-api" Nov 04 01:25:35.243784 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.243775 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8099eb0-eb29-4e6f-8715-d714104a0ff1" containerName="maas-api" Nov 04 01:25:35.249338 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.249312 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:35.260293 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.260262 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d7024f0c-openshift-default-dockercfg-bzddd\"" Nov 04 01:25:35.277692 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.277655 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2"] Nov 04 01:25:35.404440 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.404405 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istio-data\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:35.404598 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.404545 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mjnn\" (UniqueName: \"kubernetes.io/projected/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-kube-api-access-8mjnn\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:35.404598 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.404579 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-workload-certs\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:35.404697 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.404673 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istio-token\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:35.404769 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.404755 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-credential-socket\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:35.404805 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.404794 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istio-envoy\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:35.404915 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.404902 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-workload-socket\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:35.405041 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.405026 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istiod-ca-cert\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:35.405080 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.405056 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istio-podinfo\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:35.506941 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.506742 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istio-data\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:35.506941 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.506883 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mjnn\" (UniqueName: \"kubernetes.io/projected/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-kube-api-access-8mjnn\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:35.507124 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.506966 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-workload-certs\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:35.507124 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.507041 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istio-token\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:35.507124 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.507088 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-credential-socket\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:35.507278 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.507134 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istio-envoy\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:35.507278 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.507249 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-workload-socket\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:35.507407 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.507385 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istiod-ca-cert\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:35.507470 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.507432 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istio-podinfo\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:35.507861 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.507675 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-workload-certs\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:35.507861 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.507807 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-credential-socket\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:35.508242 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.508217 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-workload-socket\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:35.508242 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.508230 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istio-data\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:35.508544 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.508523 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istiod-ca-cert\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:35.510109 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.510082 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istio-envoy\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:35.510283 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.510262 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istio-podinfo\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:35.518491 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.518465 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mjnn\" (UniqueName: \"kubernetes.io/projected/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-kube-api-access-8mjnn\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:35.520716 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.520692 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istio-token\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:35.582205 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.582173 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:35.891799 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.891756 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2"] Nov 04 01:25:35.894147 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.894113 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:25:35.894266 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.894177 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:25:35.894266 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:35.894207 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:25:36.535196 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:25:36.535161 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:25:36.897939 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:36.897844 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" event={"ID":"eb3b412a-a398-4e1b-b3f9-f16a8e30c400","Type":"ContainerStarted","Data":"8cdfde760c3c036b517ae0c79f942685a232d4521e8042bcc482a5590a7f76e5"} Nov 04 01:25:36.897939 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:36.897905 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" event={"ID":"eb3b412a-a398-4e1b-b3f9-f16a8e30c400","Type":"ContainerStarted","Data":"8c7873eaf0efb69f8708d4ab1b043e3294a75a9c427cf92229c6ea854fbfd960"} Nov 04 01:25:37.583131 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:37.583095 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:37.585545 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:37.585519 7059 patch_prober.go:29] interesting pod/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 04 01:25:37.585691 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:37.585567 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" podUID="eb3b412a-a398-4e1b-b3f9-f16a8e30c400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 04 01:25:38.582772 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:38.582735 7059 patch_prober.go:29] interesting pod/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 04 01:25:38.582982 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:38.582793 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" podUID="eb3b412a-a398-4e1b-b3f9-f16a8e30c400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 04 01:25:39.584103 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:39.584070 7059 patch_prober.go:29] interesting pod/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 04 01:25:39.584486 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:39.584141 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" podUID="eb3b412a-a398-4e1b-b3f9-f16a8e30c400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 04 01:25:40.584045 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:40.584010 7059 patch_prober.go:29] interesting pod/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 04 01:25:40.584229 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:40.584074 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" podUID="eb3b412a-a398-4e1b-b3f9-f16a8e30c400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 04 01:25:41.583300 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:41.583262 7059 patch_prober.go:29] interesting pod/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 04 01:25:41.583469 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:41.583326 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" podUID="eb3b412a-a398-4e1b-b3f9-f16a8e30c400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 04 01:25:42.583462 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:42.583426 7059 patch_prober.go:29] interesting pod/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 04 01:25:42.583962 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:42.583490 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" podUID="eb3b412a-a398-4e1b-b3f9-f16a8e30c400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 04 01:25:43.590192 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:43.590158 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:43.590697 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:43.590661 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:43.591484 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:43.591457 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:43.675826 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:43.675761 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" podStartSLOduration=8.675740994 podStartE2EDuration="8.675740994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:25:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:25:36.94714001 +0000 UTC m=+1621.888988914" watchObservedRunningTime="2026-11-04 01:25:43.675740994 +0000 UTC m=+1628.617589897" Nov 04 01:25:47.313369 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:47.313173 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2"] Nov 04 01:25:47.314353 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:47.313405 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" podUID="eb3b412a-a398-4e1b-b3f9-f16a8e30c400" containerName="istio-proxy" containerID="cri-o://8cdfde760c3c036b517ae0c79f942685a232d4521e8042bcc482a5590a7f76e5" gracePeriod=30 Nov 04 01:25:52.609529 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:52.609458 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:52.753731 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:52.753695 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istio-data\") pod \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " Nov 04 01:25:52.753896 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:52.753844 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-kube-api-access-8mjnn\" (UniqueName: \"kubernetes.io/projected/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-kube-api-access-8mjnn\") pod \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " Nov 04 01:25:52.753896 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:52.753880 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istio-podinfo\") pod \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " Nov 04 01:25:52.754045 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:52.753952 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istiod-ca-cert\") pod \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " Nov 04 01:25:52.754102 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:52.754069 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istio-data" pod "eb3b412a-a398-4e1b-b3f9-f16a8e30c400" (UID: "eb3b412a-a398-4e1b-b3f9-f16a8e30c400"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:52.754157 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:52.754102 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-credential-socket\") pod \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " Nov 04 01:25:52.754212 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:52.754154 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istio-token\" (UniqueName: \"kubernetes.io/projected/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istio-token\") pod \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " Nov 04 01:25:52.754306 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:52.754287 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-workload-certs\") pod \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " Nov 04 01:25:52.754424 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:52.754324 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istio-envoy\") pod \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " Nov 04 01:25:52.754424 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:52.754378 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-credential-socket" pod "eb3b412a-a398-4e1b-b3f9-f16a8e30c400" (UID: "eb3b412a-a398-4e1b-b3f9-f16a8e30c400"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:52.754424 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:52.754390 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istiod-ca-cert" pod "eb3b412a-a398-4e1b-b3f9-f16a8e30c400" (UID: "eb3b412a-a398-4e1b-b3f9-f16a8e30c400"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:25:52.754424 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:52.754417 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-workload-socket\") pod \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\" (UID: \"eb3b412a-a398-4e1b-b3f9-f16a8e30c400\") " Nov 04 01:25:52.755525 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:52.755502 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:52.755718 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:52.755701 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:52.755818 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:52.755805 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:52.756143 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:52.756051 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-workload-socket" pod "eb3b412a-a398-4e1b-b3f9-f16a8e30c400" (UID: "eb3b412a-a398-4e1b-b3f9-f16a8e30c400"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:52.756143 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:52.756120 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-workload-certs" pod "eb3b412a-a398-4e1b-b3f9-f16a8e30c400" (UID: "eb3b412a-a398-4e1b-b3f9-f16a8e30c400"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:52.757880 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:52.757843 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istio-token" pod "eb3b412a-a398-4e1b-b3f9-f16a8e30c400" (UID: "eb3b412a-a398-4e1b-b3f9-f16a8e30c400"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:25:52.758425 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:52.758401 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istio-podinfo" pod "eb3b412a-a398-4e1b-b3f9-f16a8e30c400" (UID: "eb3b412a-a398-4e1b-b3f9-f16a8e30c400"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:25:52.758525 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:52.758406 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-kube-api-access-8mjnn" pod "eb3b412a-a398-4e1b-b3f9-f16a8e30c400" (UID: "eb3b412a-a398-4e1b-b3f9-f16a8e30c400"). InnerVolumeSpecName "kube-api-access-8mjnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:25:52.758701 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:52.758667 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istio-envoy" pod "eb3b412a-a398-4e1b-b3f9-f16a8e30c400" (UID: "eb3b412a-a398-4e1b-b3f9-f16a8e30c400"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:52.857796 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:52.857766 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mjnn\" (UniqueName: \"kubernetes.io/projected/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-kube-api-access-8mjnn\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:52.857796 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:52.857793 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:52.858032 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:52.857810 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:52.858032 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:52.857825 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:52.858032 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:52.857838 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:52.858032 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:52.857853 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb3b412a-a398-4e1b-b3f9-f16a8e30c400-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:53.361828 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:53.361792 7059 generic.go:356] "Generic (PLEG): container finished" podID="eb3b412a-a398-4e1b-b3f9-f16a8e30c400" containerID="8cdfde760c3c036b517ae0c79f942685a232d4521e8042bcc482a5590a7f76e5" exitCode=0 Nov 04 01:25:53.362017 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:53.361843 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" Nov 04 01:25:53.362017 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:53.361876 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" event={"ID":"eb3b412a-a398-4e1b-b3f9-f16a8e30c400","Type":"ContainerDied","Data":"8cdfde760c3c036b517ae0c79f942685a232d4521e8042bcc482a5590a7f76e5"} Nov 04 01:25:53.362017 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:53.361973 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2" event={"ID":"eb3b412a-a398-4e1b-b3f9-f16a8e30c400","Type":"ContainerDied","Data":"8c7873eaf0efb69f8708d4ab1b043e3294a75a9c427cf92229c6ea854fbfd960"} Nov 04 01:25:53.362017 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:53.361999 7059 scope.go:122] "RemoveContainer" containerID="8cdfde760c3c036b517ae0c79f942685a232d4521e8042bcc482a5590a7f76e5" Nov 04 01:25:53.374037 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:53.374016 7059 scope.go:122] "RemoveContainer" containerID="8cdfde760c3c036b517ae0c79f942685a232d4521e8042bcc482a5590a7f76e5" Nov 04 01:25:53.374355 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:25:53.374337 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cdfde760c3c036b517ae0c79f942685a232d4521e8042bcc482a5590a7f76e5\": container with ID starting with 8cdfde760c3c036b517ae0c79f942685a232d4521e8042bcc482a5590a7f76e5 not found: ID does not exist" containerID="8cdfde760c3c036b517ae0c79f942685a232d4521e8042bcc482a5590a7f76e5" Nov 04 01:25:53.374424 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:53.374363 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cdfde760c3c036b517ae0c79f942685a232d4521e8042bcc482a5590a7f76e5"} err="failed to get container status \"8cdfde760c3c036b517ae0c79f942685a232d4521e8042bcc482a5590a7f76e5\": rpc error: code = NotFound desc = could not find container \"8cdfde760c3c036b517ae0c79f942685a232d4521e8042bcc482a5590a7f76e5\": container with ID starting with 8cdfde760c3c036b517ae0c79f942685a232d4521e8042bcc482a5590a7f76e5 not found: ID does not exist" Nov 04 01:25:53.385311 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:53.385290 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2"] Nov 04 01:25:53.404830 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:53.404791 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-6pln2"] Nov 04 01:25:55.180543 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:25:55.180510 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb3b412a-a398-4e1b-b3f9-f16a8e30c400" path="/var/lib/kubelet/pods/eb3b412a-a398-4e1b-b3f9-f16a8e30c400/volumes" Nov 04 01:26:06.587951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:06.587595 7059 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:26:06.587951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:06.587675 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" podUID="b59977fd-6792-40e9-9070-1f04beb3f0bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:06.825892 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:26:06.824235 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:26:27.087466 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:27.087433 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2"] Nov 04 01:26:27.088218 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:27.088204 7059 cpu_manager.go:414] "containerMap: removing container" podUID="eb3b412a-a398-4e1b-b3f9-f16a8e30c400" containerName="istio-proxy" Nov 04 01:26:27.088279 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:27.088220 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb3b412a-a398-4e1b-b3f9-f16a8e30c400" containerName="istio-proxy" Nov 04 01:26:27.088388 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:27.088379 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb3b412a-a398-4e1b-b3f9-f16a8e30c400" containerName="istio-proxy" Nov 04 01:26:27.092197 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:27.092177 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2" Nov 04 01:26:27.109444 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:27.109422 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-dfcebf-serving-cert\"" Nov 04 01:26:27.121305 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:27.121281 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2"] Nov 04 01:26:27.202120 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:27.202085 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4630234e-0999-43f6-9221-007ed3f80001-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2\" (UID: \"4630234e-0999-43f6-9221-007ed3f80001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2" Nov 04 01:26:27.202120 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:27.202118 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4630234e-0999-43f6-9221-007ed3f80001-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2\" (UID: \"4630234e-0999-43f6-9221-007ed3f80001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2" Nov 04 01:26:27.202342 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:27.202249 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4630234e-0999-43f6-9221-007ed3f80001-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2\" (UID: \"4630234e-0999-43f6-9221-007ed3f80001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2" Nov 04 01:26:27.202488 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:27.202467 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-975w2\" (UniqueName: \"kubernetes.io/projected/4630234e-0999-43f6-9221-007ed3f80001-kube-api-access-975w2\") pod \"maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2\" (UID: \"4630234e-0999-43f6-9221-007ed3f80001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2" Nov 04 01:26:27.304041 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:27.304008 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-975w2\" (UniqueName: \"kubernetes.io/projected/4630234e-0999-43f6-9221-007ed3f80001-kube-api-access-975w2\") pod \"maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2\" (UID: \"4630234e-0999-43f6-9221-007ed3f80001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2" Nov 04 01:26:27.304320 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:27.304295 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4630234e-0999-43f6-9221-007ed3f80001-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2\" (UID: \"4630234e-0999-43f6-9221-007ed3f80001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2" Nov 04 01:26:27.304398 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:27.304339 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4630234e-0999-43f6-9221-007ed3f80001-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2\" (UID: \"4630234e-0999-43f6-9221-007ed3f80001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2" Nov 04 01:26:27.304459 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:27.304410 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4630234e-0999-43f6-9221-007ed3f80001-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2\" (UID: \"4630234e-0999-43f6-9221-007ed3f80001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2" Nov 04 01:26:27.305236 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:27.305211 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4630234e-0999-43f6-9221-007ed3f80001-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2\" (UID: \"4630234e-0999-43f6-9221-007ed3f80001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2" Nov 04 01:26:27.306882 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:27.306860 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4630234e-0999-43f6-9221-007ed3f80001-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2\" (UID: \"4630234e-0999-43f6-9221-007ed3f80001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2" Nov 04 01:26:27.306882 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:27.306873 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4630234e-0999-43f6-9221-007ed3f80001-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2\" (UID: \"4630234e-0999-43f6-9221-007ed3f80001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2" Nov 04 01:26:27.317955 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:27.317899 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-975w2\" (UniqueName: \"kubernetes.io/projected/4630234e-0999-43f6-9221-007ed3f80001-kube-api-access-975w2\") pod \"maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2\" (UID: \"4630234e-0999-43f6-9221-007ed3f80001\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2" Nov 04 01:26:27.406366 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:27.406300 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2" Nov 04 01:26:27.704849 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:26:27.704814 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4630234e_0999_43f6_9221_007ed3f80001.slice/crio-f3bac79b43cb06a1125e1c7566c0eb3bade68238e77ba73e52c39f3091be1d36 WatchSource:0}: Error finding container f3bac79b43cb06a1125e1c7566c0eb3bade68238e77ba73e52c39f3091be1d36: Status 404 returned error can't find the container with id f3bac79b43cb06a1125e1c7566c0eb3bade68238e77ba73e52c39f3091be1d36 Nov 04 01:26:27.706941 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:27.706886 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2"] Nov 04 01:26:28.152499 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:28.152466 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2" event={"ID":"4630234e-0999-43f6-9221-007ed3f80001","Type":"ContainerStarted","Data":"f3bac79b43cb06a1125e1c7566c0eb3bade68238e77ba73e52c39f3091be1d36"} Nov 04 01:26:29.164529 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:29.164493 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2" event={"ID":"4630234e-0999-43f6-9221-007ed3f80001","Type":"ContainerStarted","Data":"7f9ad845d074fe5ec22555a7939d0416b5b317108fdfec0663979b3321a8227b"} Nov 04 01:26:29.165020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:29.164554 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2" Nov 04 01:26:29.199464 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:29.199405 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2" podStartSLOduration=1.768128199 podStartE2EDuration="2.199383289s" totalImagesPullingTime="431.25509ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:29.185771912 +0000 UTC m=+1674.127620816" watchObservedRunningTime="2026-11-04 01:26:29.199383289 +0000 UTC m=+1674.141232193" Nov 04 01:26:35.181127 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:35.181096 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2" Nov 04 01:26:36.581881 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:26:36.581843 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:26:53.508274 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:53.508242 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v"] Nov 04 01:26:53.515114 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:53.515083 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v" Nov 04 01:26:53.530884 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:53.530854 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-016d3d-serving-cert\"" Nov 04 01:26:53.536200 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:53.536169 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v"] Nov 04 01:26:53.573589 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:53.573558 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f303474f-5ca6-484f-92de-b8080ba7bbc5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v\" (UID: \"f303474f-5ca6-484f-92de-b8080ba7bbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v" Nov 04 01:26:53.573742 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:53.573621 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f303474f-5ca6-484f-92de-b8080ba7bbc5-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v\" (UID: \"f303474f-5ca6-484f-92de-b8080ba7bbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v" Nov 04 01:26:53.573822 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:53.573753 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfn52\" (UniqueName: \"kubernetes.io/projected/f303474f-5ca6-484f-92de-b8080ba7bbc5-kube-api-access-mfn52\") pod \"maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v\" (UID: \"f303474f-5ca6-484f-92de-b8080ba7bbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v" Nov 04 01:26:53.573916 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:53.573898 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f303474f-5ca6-484f-92de-b8080ba7bbc5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v\" (UID: \"f303474f-5ca6-484f-92de-b8080ba7bbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v" Nov 04 01:26:53.676451 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:53.676407 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f303474f-5ca6-484f-92de-b8080ba7bbc5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v\" (UID: \"f303474f-5ca6-484f-92de-b8080ba7bbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v" Nov 04 01:26:53.676671 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:53.676480 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f303474f-5ca6-484f-92de-b8080ba7bbc5-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v\" (UID: \"f303474f-5ca6-484f-92de-b8080ba7bbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v" Nov 04 01:26:53.676671 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:53.676547 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfn52\" (UniqueName: \"kubernetes.io/projected/f303474f-5ca6-484f-92de-b8080ba7bbc5-kube-api-access-mfn52\") pod \"maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v\" (UID: \"f303474f-5ca6-484f-92de-b8080ba7bbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v" Nov 04 01:26:53.676806 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:53.676703 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f303474f-5ca6-484f-92de-b8080ba7bbc5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v\" (UID: \"f303474f-5ca6-484f-92de-b8080ba7bbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v" Nov 04 01:26:53.679079 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:53.679046 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f303474f-5ca6-484f-92de-b8080ba7bbc5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v\" (UID: \"f303474f-5ca6-484f-92de-b8080ba7bbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v" Nov 04 01:26:53.681142 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:53.681096 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f303474f-5ca6-484f-92de-b8080ba7bbc5-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v\" (UID: \"f303474f-5ca6-484f-92de-b8080ba7bbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v" Nov 04 01:26:53.688954 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:53.684820 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f303474f-5ca6-484f-92de-b8080ba7bbc5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v\" (UID: \"f303474f-5ca6-484f-92de-b8080ba7bbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v" Nov 04 01:26:53.692503 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:53.692471 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfn52\" (UniqueName: \"kubernetes.io/projected/f303474f-5ca6-484f-92de-b8080ba7bbc5-kube-api-access-mfn52\") pod \"maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v\" (UID: \"f303474f-5ca6-484f-92de-b8080ba7bbc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v" Nov 04 01:26:53.844975 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:53.844861 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v" Nov 04 01:26:54.158766 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:54.157906 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v"] Nov 04 01:26:54.158766 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:26:54.158195 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf303474f_5ca6_484f_92de_b8080ba7bbc5.slice/crio-b1fcd66caf2826530be426758a928156e8e472158958c39e861385c80aeb05ca WatchSource:0}: Error finding container b1fcd66caf2826530be426758a928156e8e472158958c39e861385c80aeb05ca: Status 404 returned error can't find the container with id b1fcd66caf2826530be426758a928156e8e472158958c39e861385c80aeb05ca Nov 04 01:26:54.445859 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:54.445822 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v" event={"ID":"f303474f-5ca6-484f-92de-b8080ba7bbc5","Type":"ContainerStarted","Data":"b1fcd66caf2826530be426758a928156e8e472158958c39e861385c80aeb05ca"} Nov 04 01:26:55.455574 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:55.455542 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v" event={"ID":"f303474f-5ca6-484f-92de-b8080ba7bbc5","Type":"ContainerStarted","Data":"ee87dbe2789799d1dee26588d2d6eb2c898e5f537a2f0cca8031a2a895907870"} Nov 04 01:26:55.456101 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:55.455683 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v" Nov 04 01:26:55.477477 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:26:55.477433 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v" podStartSLOduration=1.9971647259999998 podStartE2EDuration="2.477422537s" totalImagesPullingTime="480.257811ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:55.474290783 +0000 UTC m=+1700.416139698" watchObservedRunningTime="2026-11-04 01:26:55.477422537 +0000 UTC m=+1700.419271469" Nov 04 01:27:01.487949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:01.486155 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v" Nov 04 01:27:06.491105 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:27:06.491065 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:27:12.721346 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:12.721307 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m"] Nov 04 01:27:12.728495 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:12.728471 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m" Nov 04 01:27:12.738427 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:12.738395 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-ef6cc9-serving-cert\"" Nov 04 01:27:12.740468 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:12.740443 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m"] Nov 04 01:27:12.794690 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:12.794661 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg64g\" (UniqueName: \"kubernetes.io/projected/fc89af7d-03b6-458b-9265-fb7d71459d81-kube-api-access-zg64g\") pod \"maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m\" (UID: \"fc89af7d-03b6-458b-9265-fb7d71459d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m" Nov 04 01:27:12.794834 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:12.794814 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc89af7d-03b6-458b-9265-fb7d71459d81-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m\" (UID: \"fc89af7d-03b6-458b-9265-fb7d71459d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m" Nov 04 01:27:12.794891 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:12.794861 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc89af7d-03b6-458b-9265-fb7d71459d81-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m\" (UID: \"fc89af7d-03b6-458b-9265-fb7d71459d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m" Nov 04 01:27:12.794891 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:12.794883 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc89af7d-03b6-458b-9265-fb7d71459d81-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m\" (UID: \"fc89af7d-03b6-458b-9265-fb7d71459d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m" Nov 04 01:27:12.896325 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:12.896289 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg64g\" (UniqueName: \"kubernetes.io/projected/fc89af7d-03b6-458b-9265-fb7d71459d81-kube-api-access-zg64g\") pod \"maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m\" (UID: \"fc89af7d-03b6-458b-9265-fb7d71459d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m" Nov 04 01:27:12.896471 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:12.896435 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc89af7d-03b6-458b-9265-fb7d71459d81-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m\" (UID: \"fc89af7d-03b6-458b-9265-fb7d71459d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m" Nov 04 01:27:12.896530 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:12.896487 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc89af7d-03b6-458b-9265-fb7d71459d81-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m\" (UID: \"fc89af7d-03b6-458b-9265-fb7d71459d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m" Nov 04 01:27:12.896530 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:12.896504 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc89af7d-03b6-458b-9265-fb7d71459d81-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m\" (UID: \"fc89af7d-03b6-458b-9265-fb7d71459d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m" Nov 04 01:27:12.897225 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:12.897196 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc89af7d-03b6-458b-9265-fb7d71459d81-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m\" (UID: \"fc89af7d-03b6-458b-9265-fb7d71459d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m" Nov 04 01:27:12.899160 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:12.899138 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc89af7d-03b6-458b-9265-fb7d71459d81-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m\" (UID: \"fc89af7d-03b6-458b-9265-fb7d71459d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m" Nov 04 01:27:12.899456 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:12.899436 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc89af7d-03b6-458b-9265-fb7d71459d81-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m\" (UID: \"fc89af7d-03b6-458b-9265-fb7d71459d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m" Nov 04 01:27:12.904857 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:12.904833 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg64g\" (UniqueName: \"kubernetes.io/projected/fc89af7d-03b6-458b-9265-fb7d71459d81-kube-api-access-zg64g\") pod \"maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m\" (UID: \"fc89af7d-03b6-458b-9265-fb7d71459d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m" Nov 04 01:27:13.054201 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:13.054116 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m" Nov 04 01:27:13.368963 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:13.368745 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m"] Nov 04 01:27:13.624674 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:13.624596 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m" event={"ID":"fc89af7d-03b6-458b-9265-fb7d71459d81","Type":"ContainerStarted","Data":"61de49cdfc87724d129d42288bfe73ac4afd7e8958306192b6b6ecad2e31e484"} Nov 04 01:27:14.637681 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:14.637626 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m" event={"ID":"fc89af7d-03b6-458b-9265-fb7d71459d81","Type":"ContainerStarted","Data":"6be37a4ae275545b6e3afe12b537bc242fd2f90fbe2926f29f6bb8adb642af44"} Nov 04 01:27:14.638734 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:14.638702 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m" Nov 04 01:27:14.659937 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:14.659846 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m" podStartSLOduration=2.197240413 podStartE2EDuration="2.65983226s" totalImagesPullingTime="462.591847ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:14.656431094 +0000 UTC m=+1719.598279996" watchObservedRunningTime="2026-11-04 01:27:14.65983226 +0000 UTC m=+1719.601681163" Nov 04 01:27:21.661610 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:21.661579 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m" Nov 04 01:27:31.625161 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:31.625127 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3888c434-7c6f7d487-56x2x"] Nov 04 01:27:31.635140 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:31.635111 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3888c434-7c6f7d487-56x2x" Nov 04 01:27:31.641330 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:31.641288 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3888c434-7c6f7d487-56x2x"] Nov 04 01:27:31.644175 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:31.644151 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3888c434-serving-cert\"" Nov 04 01:27:31.700747 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:31.700605 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b6d9ff3-bd4b-464c-851e-ac16a107ddda-openshift-service-ca\") pod \"maas-api-e2e-mt-3888c434-7c6f7d487-56x2x\" (UID: \"8b6d9ff3-bd4b-464c-851e-ac16a107ddda\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3888c434-7c6f7d487-56x2x" Nov 04 01:27:31.700747 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:31.700702 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b6d9ff3-bd4b-464c-851e-ac16a107ddda-maas-api-tls\") pod \"maas-api-e2e-mt-3888c434-7c6f7d487-56x2x\" (UID: \"8b6d9ff3-bd4b-464c-851e-ac16a107ddda\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3888c434-7c6f7d487-56x2x" Nov 04 01:27:31.700747 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:31.700747 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b6d9ff3-bd4b-464c-851e-ac16a107ddda-metrics-certs\") pod \"maas-api-e2e-mt-3888c434-7c6f7d487-56x2x\" (UID: \"8b6d9ff3-bd4b-464c-851e-ac16a107ddda\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3888c434-7c6f7d487-56x2x" Nov 04 01:27:31.701096 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:31.701076 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j77rt\" (UniqueName: \"kubernetes.io/projected/8b6d9ff3-bd4b-464c-851e-ac16a107ddda-kube-api-access-j77rt\") pod \"maas-api-e2e-mt-3888c434-7c6f7d487-56x2x\" (UID: \"8b6d9ff3-bd4b-464c-851e-ac16a107ddda\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3888c434-7c6f7d487-56x2x" Nov 04 01:27:31.803198 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:31.803164 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j77rt\" (UniqueName: \"kubernetes.io/projected/8b6d9ff3-bd4b-464c-851e-ac16a107ddda-kube-api-access-j77rt\") pod \"maas-api-e2e-mt-3888c434-7c6f7d487-56x2x\" (UID: \"8b6d9ff3-bd4b-464c-851e-ac16a107ddda\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3888c434-7c6f7d487-56x2x" Nov 04 01:27:31.803372 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:31.803346 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b6d9ff3-bd4b-464c-851e-ac16a107ddda-openshift-service-ca\") pod \"maas-api-e2e-mt-3888c434-7c6f7d487-56x2x\" (UID: \"8b6d9ff3-bd4b-464c-851e-ac16a107ddda\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3888c434-7c6f7d487-56x2x" Nov 04 01:27:31.803437 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:31.803380 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b6d9ff3-bd4b-464c-851e-ac16a107ddda-maas-api-tls\") pod \"maas-api-e2e-mt-3888c434-7c6f7d487-56x2x\" (UID: \"8b6d9ff3-bd4b-464c-851e-ac16a107ddda\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3888c434-7c6f7d487-56x2x" Nov 04 01:27:31.803437 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:31.803402 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b6d9ff3-bd4b-464c-851e-ac16a107ddda-metrics-certs\") pod \"maas-api-e2e-mt-3888c434-7c6f7d487-56x2x\" (UID: \"8b6d9ff3-bd4b-464c-851e-ac16a107ddda\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3888c434-7c6f7d487-56x2x" Nov 04 01:27:31.804059 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:31.804035 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b6d9ff3-bd4b-464c-851e-ac16a107ddda-openshift-service-ca\") pod \"maas-api-e2e-mt-3888c434-7c6f7d487-56x2x\" (UID: \"8b6d9ff3-bd4b-464c-851e-ac16a107ddda\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3888c434-7c6f7d487-56x2x" Nov 04 01:27:31.806283 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:31.806261 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b6d9ff3-bd4b-464c-851e-ac16a107ddda-maas-api-tls\") pod \"maas-api-e2e-mt-3888c434-7c6f7d487-56x2x\" (UID: \"8b6d9ff3-bd4b-464c-851e-ac16a107ddda\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3888c434-7c6f7d487-56x2x" Nov 04 01:27:31.806396 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:31.806359 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b6d9ff3-bd4b-464c-851e-ac16a107ddda-metrics-certs\") pod \"maas-api-e2e-mt-3888c434-7c6f7d487-56x2x\" (UID: \"8b6d9ff3-bd4b-464c-851e-ac16a107ddda\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3888c434-7c6f7d487-56x2x" Nov 04 01:27:31.812622 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:31.812599 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j77rt\" (UniqueName: \"kubernetes.io/projected/8b6d9ff3-bd4b-464c-851e-ac16a107ddda-kube-api-access-j77rt\") pod \"maas-api-e2e-mt-3888c434-7c6f7d487-56x2x\" (UID: \"8b6d9ff3-bd4b-464c-851e-ac16a107ddda\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3888c434-7c6f7d487-56x2x" Nov 04 01:27:31.958073 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:31.958038 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3888c434-7c6f7d487-56x2x" Nov 04 01:27:32.262128 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:27:32.262100 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b6d9ff3_bd4b_464c_851e_ac16a107ddda.slice/crio-67dc8e30b0871e163356188330c5d972db9ca8fb472c7ac14c23a5d485ba4dd9 WatchSource:0}: Error finding container 67dc8e30b0871e163356188330c5d972db9ca8fb472c7ac14c23a5d485ba4dd9: Status 404 returned error can't find the container with id 67dc8e30b0871e163356188330c5d972db9ca8fb472c7ac14c23a5d485ba4dd9 Nov 04 01:27:32.263772 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:32.263744 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3888c434-7c6f7d487-56x2x"] Nov 04 01:27:32.974020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:32.971622 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3888c434-7c6f7d487-56x2x" event={"ID":"8b6d9ff3-bd4b-464c-851e-ac16a107ddda","Type":"ContainerStarted","Data":"d2a813dac09aa318cb8a5ceca0c5c49672205bf6d486f9a7c928e881236851e5"} Nov 04 01:27:32.974020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:32.971745 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3888c434-7c6f7d487-56x2x" event={"ID":"8b6d9ff3-bd4b-464c-851e-ac16a107ddda","Type":"ContainerStarted","Data":"67dc8e30b0871e163356188330c5d972db9ca8fb472c7ac14c23a5d485ba4dd9"} Nov 04 01:27:32.974020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:32.973013 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3888c434-7c6f7d487-56x2x" Nov 04 01:27:32.994977 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:32.994903 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3888c434-7c6f7d487-56x2x" podStartSLOduration=1.572948633 podStartE2EDuration="1.994883569s" totalImagesPullingTime="421.934936ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:32.990087221 +0000 UTC m=+1737.931936124" watchObservedRunningTime="2026-11-04 01:27:32.994883569 +0000 UTC m=+1737.936732474" Nov 04 01:27:36.499033 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:27:36.498912 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:27:40.007721 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:40.007689 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3888c434-7c6f7d487-56x2x" Nov 04 01:27:49.499670 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:49.499636 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2"] Nov 04 01:27:49.516808 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:49.516774 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2"] Nov 04 01:27:49.516993 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:49.516901 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2" Nov 04 01:27:49.523171 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:49.523147 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-2fff32-serving-cert\"" Nov 04 01:27:49.619291 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:49.619262 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdb4a298-c600-4362-97a0-8780f0e48532-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2\" (UID: \"fdb4a298-c600-4362-97a0-8780f0e48532\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2" Nov 04 01:27:49.619442 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:49.619304 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdb4a298-c600-4362-97a0-8780f0e48532-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2\" (UID: \"fdb4a298-c600-4362-97a0-8780f0e48532\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2" Nov 04 01:27:49.619494 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:49.619438 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxhtm\" (UniqueName: \"kubernetes.io/projected/fdb4a298-c600-4362-97a0-8780f0e48532-kube-api-access-xxhtm\") pod \"maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2\" (UID: \"fdb4a298-c600-4362-97a0-8780f0e48532\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2" Nov 04 01:27:49.619760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:49.619741 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdb4a298-c600-4362-97a0-8780f0e48532-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2\" (UID: \"fdb4a298-c600-4362-97a0-8780f0e48532\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2" Nov 04 01:27:49.721337 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:49.721235 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdb4a298-c600-4362-97a0-8780f0e48532-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2\" (UID: \"fdb4a298-c600-4362-97a0-8780f0e48532\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2" Nov 04 01:27:49.721519 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:49.721462 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxhtm\" (UniqueName: \"kubernetes.io/projected/fdb4a298-c600-4362-97a0-8780f0e48532-kube-api-access-xxhtm\") pod \"maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2\" (UID: \"fdb4a298-c600-4362-97a0-8780f0e48532\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2" Nov 04 01:27:49.721821 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:49.721793 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdb4a298-c600-4362-97a0-8780f0e48532-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2\" (UID: \"fdb4a298-c600-4362-97a0-8780f0e48532\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2" Nov 04 01:27:49.721970 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:49.721901 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdb4a298-c600-4362-97a0-8780f0e48532-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2\" (UID: \"fdb4a298-c600-4362-97a0-8780f0e48532\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2" Nov 04 01:27:49.722703 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:49.722642 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdb4a298-c600-4362-97a0-8780f0e48532-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2\" (UID: \"fdb4a298-c600-4362-97a0-8780f0e48532\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2" Nov 04 01:27:49.724800 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:49.724777 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdb4a298-c600-4362-97a0-8780f0e48532-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2\" (UID: \"fdb4a298-c600-4362-97a0-8780f0e48532\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2" Nov 04 01:27:49.725355 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:49.725333 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdb4a298-c600-4362-97a0-8780f0e48532-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2\" (UID: \"fdb4a298-c600-4362-97a0-8780f0e48532\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2" Nov 04 01:27:49.730436 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:49.730414 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxhtm\" (UniqueName: \"kubernetes.io/projected/fdb4a298-c600-4362-97a0-8780f0e48532-kube-api-access-xxhtm\") pod \"maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2\" (UID: \"fdb4a298-c600-4362-97a0-8780f0e48532\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2" Nov 04 01:27:49.846009 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:49.843224 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2" Nov 04 01:27:50.358335 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:50.355463 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2"] Nov 04 01:27:50.360631 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:27:50.359944 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdb4a298_c600_4362_97a0_8780f0e48532.slice/crio-c61b828fa4920971e05d6d2fce0ef828e5a7b20a3013a6c737effa16940fe818 WatchSource:0}: Error finding container c61b828fa4920971e05d6d2fce0ef828e5a7b20a3013a6c737effa16940fe818: Status 404 returned error can't find the container with id c61b828fa4920971e05d6d2fce0ef828e5a7b20a3013a6c737effa16940fe818 Nov 04 01:27:51.146372 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:51.146276 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2" event={"ID":"fdb4a298-c600-4362-97a0-8780f0e48532","Type":"ContainerStarted","Data":"e2709bd90ef774b8ed4bb31f8576064377c96d03cf2527a7eab03a9fc5bb1bff"} Nov 04 01:27:51.146730 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:51.146498 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2" Nov 04 01:27:51.146730 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:51.146527 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2" event={"ID":"fdb4a298-c600-4362-97a0-8780f0e48532","Type":"ContainerStarted","Data":"c61b828fa4920971e05d6d2fce0ef828e5a7b20a3013a6c737effa16940fe818"} Nov 04 01:27:51.172579 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:51.172506 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2" podStartSLOduration=1.746442117 podStartE2EDuration="2.172492734s" totalImagesPullingTime="426.050617ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:51.166697939 +0000 UTC m=+1756.108546854" watchObservedRunningTime="2026-11-04 01:27:51.172492734 +0000 UTC m=+1756.114341638" Nov 04 01:27:57.166089 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:27:57.166059 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2" Nov 04 01:28:00.722428 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:00.722390 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z"] Nov 04 01:28:00.728788 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:00.728759 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:00.736426 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:00.736393 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-ef6cc9-openshift-default-dockercfg-m7xbv\"" Nov 04 01:28:00.745443 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:00.745413 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z"] Nov 04 01:28:00.900088 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:00.900045 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-workload-certs\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:00.900242 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:00.900190 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-credential-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:00.900291 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:00.900258 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftlzm\" (UniqueName: \"kubernetes.io/projected/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-kube-api-access-ftlzm\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:00.900383 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:00.900367 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istio-podinfo\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:00.900510 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:00.900489 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istio-envoy\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:00.900571 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:00.900560 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istio-data\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:00.900605 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:00.900583 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istiod-ca-cert\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:00.900775 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:00.900748 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-workload-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:00.900965 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:00.900850 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istio-token\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:01.002829 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:01.002725 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-workload-certs\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:01.003000 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:01.002835 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-credential-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:01.003000 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:01.002863 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftlzm\" (UniqueName: \"kubernetes.io/projected/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-kube-api-access-ftlzm\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:01.003000 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:01.002970 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istio-podinfo\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:01.003181 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:01.003090 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istio-envoy\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:01.003241 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:01.003194 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-workload-certs\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:01.003348 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:01.003323 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-credential-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:01.003579 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:01.003556 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istio-data\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:01.003698 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:01.003593 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istiod-ca-cert\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:01.003698 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:01.003650 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-workload-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:01.004055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:01.004030 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istio-data\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:01.004149 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:01.004086 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istio-token\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:01.004213 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:01.004193 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-workload-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:01.004447 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:01.004422 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istiod-ca-cert\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:01.005750 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:01.005719 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istio-envoy\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:01.005995 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:01.005973 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istio-podinfo\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:01.011971 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:01.011742 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftlzm\" (UniqueName: \"kubernetes.io/projected/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-kube-api-access-ftlzm\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:01.012411 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:01.012389 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istio-token\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:01.051560 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:01.051534 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:01.386005 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:01.382684 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z"] Nov 04 01:28:01.407503 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:01.407458 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:28:01.407636 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:01.407549 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:28:01.407636 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:01.407583 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:28:02.264636 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:02.264602 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" event={"ID":"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee","Type":"ContainerStarted","Data":"9ca62b5f6004c1fd875d6f86a094467c63563609f5130cad39e3064338b9346f"} Nov 04 01:28:02.265082 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:02.264658 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" event={"ID":"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee","Type":"ContainerStarted","Data":"cbd60d368592a9bfc522def33537708fe55607826a51e0ebdb6ba72ed434d3e6"} Nov 04 01:28:02.310281 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:02.310229 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podStartSLOduration=2.310213959 podStartE2EDuration="2.310213959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:02.308170094 +0000 UTC m=+1767.250019000" watchObservedRunningTime="2026-11-04 01:28:02.310213959 +0000 UTC m=+1767.252062864" Nov 04 01:28:03.051699 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:03.051665 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:03.053753 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:03.053725 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 01:28:03.053900 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:03.053790 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 01:28:04.052518 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:04.052483 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 01:28:04.053034 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:04.052540 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 01:28:05.052040 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:05.052006 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 01:28:05.052239 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:05.052068 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 01:28:06.052530 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:06.052493 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 01:28:06.052986 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:06.052558 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 01:28:06.484985 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:28:06.484388 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:28:07.052657 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:07.052626 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 01:28:07.055178 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:07.052681 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 01:28:08.052139 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:08.052107 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 01:28:08.052316 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:08.052161 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 01:28:09.052360 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:09.052321 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 01:28:09.052766 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:09.052397 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 01:28:10.051934 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:10.051900 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 01:28:10.052117 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:10.051973 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 01:28:11.052111 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:11.052077 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 01:28:11.052628 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:11.052144 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 01:28:12.052105 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:12.052069 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 01:28:12.052307 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:12.052131 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 01:28:13.052152 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:13.052113 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 01:28:13.052319 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:13.052169 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 01:28:14.052032 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:14.052000 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 01:28:14.052227 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:14.052052 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 01:28:15.052482 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:15.052451 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 01:28:15.052861 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:15.052509 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 01:28:15.752190 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:15.752135 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww"] Nov 04 01:28:15.758126 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:15.758100 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:28:15.785229 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:15.785202 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww"] Nov 04 01:28:15.853819 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:15.853783 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-credential-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:28:15.854029 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:15.853837 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-workload-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:28:15.854029 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:15.853977 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-workload-certs\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:28:15.854029 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:15.854011 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91e372a-c89d-41f6-a700-8c41a419a293-istiod-ca-cert\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:28:15.854249 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:15.854235 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggn7j\" (UniqueName: \"kubernetes.io/projected/d91e372a-c89d-41f6-a700-8c41a419a293-kube-api-access-ggn7j\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:28:15.854296 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:15.854259 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-istio-envoy\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:28:15.854343 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:15.854308 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91e372a-c89d-41f6-a700-8c41a419a293-istio-podinfo\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:28:15.854415 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:15.854401 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-istio-data\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:28:15.854452 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:15.854439 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91e372a-c89d-41f6-a700-8c41a419a293-istio-token\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:28:15.955624 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:15.955593 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-credential-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:28:15.955811 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:15.955651 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-workload-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:28:15.955811 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:15.955700 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-workload-certs\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:28:15.955811 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:15.955737 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91e372a-c89d-41f6-a700-8c41a419a293-istiod-ca-cert\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:28:15.956043 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:15.956019 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggn7j\" (UniqueName: \"kubernetes.io/projected/d91e372a-c89d-41f6-a700-8c41a419a293-kube-api-access-ggn7j\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:28:15.956179 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:15.956058 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-istio-envoy\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:28:15.956179 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:15.956085 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-credential-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:28:15.956179 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:15.956123 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91e372a-c89d-41f6-a700-8c41a419a293-istio-podinfo\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:28:15.956339 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:15.956203 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-istio-data\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:28:15.956339 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:15.956210 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-workload-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:28:15.956339 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:15.956273 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91e372a-c89d-41f6-a700-8c41a419a293-istio-token\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:28:15.956481 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:15.956455 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-workload-certs\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:28:15.956710 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:15.956689 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-istio-data\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:28:15.957074 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:15.956877 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91e372a-c89d-41f6-a700-8c41a419a293-istiod-ca-cert\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:28:15.958978 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:15.958945 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-istio-envoy\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:28:15.959060 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:15.959013 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91e372a-c89d-41f6-a700-8c41a419a293-istio-podinfo\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:28:15.967296 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:15.967276 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91e372a-c89d-41f6-a700-8c41a419a293-istio-token\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:28:15.967758 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:15.967734 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggn7j\" (UniqueName: \"kubernetes.io/projected/d91e372a-c89d-41f6-a700-8c41a419a293-kube-api-access-ggn7j\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:28:16.051981 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:16.051880 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:16.052146 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:16.052126 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 01:28:16.052203 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:16.052183 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 01:28:16.092695 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:16.092669 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:28:16.394269 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:16.394240 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww"] Nov 04 01:28:16.398452 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:16.398410 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:28:16.398563 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:16.398494 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:28:16.398563 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:16.398543 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:28:17.052069 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:17.052025 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 01:28:17.052246 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:17.052081 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 01:28:17.393519 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:17.393434 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" event={"ID":"d91e372a-c89d-41f6-a700-8c41a419a293","Type":"ContainerStarted","Data":"44c0dabb823f6fd42a90bf0f8aec1370e34f941f0e9d82ba5c5b39f64a493648"} Nov 04 01:28:17.393519 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:17.393499 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" event={"ID":"d91e372a-c89d-41f6-a700-8c41a419a293","Type":"ContainerStarted","Data":"cb81ee85762c8878a79d5e19b17586d44f7d56df8261998455e958f0daf6e5ad"} Nov 04 01:28:17.426864 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:17.426785 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" podStartSLOduration=2.426767554 podStartE2EDuration="2.426767554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:17.424810707 +0000 UTC m=+1782.366659611" watchObservedRunningTime="2026-11-04 01:28:17.426767554 +0000 UTC m=+1782.368616457" Nov 04 01:28:18.052402 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:18.052361 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 01:28:18.052600 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:18.052449 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 01:28:18.092810 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:18.092779 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:28:18.094379 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:18.094352 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 04 01:28:18.094487 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:18.094427 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" podUID="d91e372a-c89d-41f6-a700-8c41a419a293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 04 01:28:19.052425 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:19.052390 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 01:28:19.052809 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:19.052455 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 01:28:19.093954 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:19.093913 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 04 01:28:19.094096 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:19.093970 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" podUID="d91e372a-c89d-41f6-a700-8c41a419a293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 04 01:28:20.052139 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:20.052106 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 01:28:20.052331 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:20.052163 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 01:28:20.093503 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:20.093473 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 04 01:28:20.093859 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:20.093528 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" podUID="d91e372a-c89d-41f6-a700-8c41a419a293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 04 01:28:21.054027 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:21.053979 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 01:28:21.054228 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:21.054049 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 01:28:21.093799 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:21.093740 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 04 01:28:21.094331 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:21.093815 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" podUID="d91e372a-c89d-41f6-a700-8c41a419a293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 04 01:28:22.052481 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:22.052442 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 01:28:22.052663 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:22.052494 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 01:28:22.093200 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:22.093165 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 04 01:28:22.093362 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:22.093230 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" podUID="d91e372a-c89d-41f6-a700-8c41a419a293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 04 01:28:23.052723 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:23.052690 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 01:28:23.053210 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:23.052747 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 01:28:23.093899 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:23.093867 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 04 01:28:23.094121 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:23.093958 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" podUID="d91e372a-c89d-41f6-a700-8c41a419a293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 04 01:28:24.052713 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:24.052681 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 01:28:24.052882 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:24.052743 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 01:28:24.093456 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:24.093428 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 04 01:28:24.093580 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:24.093474 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" podUID="d91e372a-c89d-41f6-a700-8c41a419a293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 04 01:28:25.051811 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:25.051778 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 01:28:25.052023 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:25.051831 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 01:28:25.093750 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:25.093719 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 04 01:28:25.094113 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:25.093768 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" podUID="d91e372a-c89d-41f6-a700-8c41a419a293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 04 01:28:26.052008 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:26.051966 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 01:28:26.052206 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:26.052026 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 01:28:26.093792 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:26.093760 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 04 01:28:26.094194 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:26.093799 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" podUID="d91e372a-c89d-41f6-a700-8c41a419a293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 04 01:28:27.052403 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:27.052362 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 01:28:27.052572 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:27.052416 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 01:28:27.093013 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:27.092981 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 04 01:28:27.093142 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:27.093031 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" podUID="d91e372a-c89d-41f6-a700-8c41a419a293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 04 01:28:28.051857 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:28.051823 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 01:28:28.052279 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:28.051883 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 01:28:28.093634 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:28.093604 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 04 01:28:28.093776 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:28.093652 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" podUID="d91e372a-c89d-41f6-a700-8c41a419a293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 04 01:28:29.052634 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:29.052603 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 01:28:29.053070 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:29.052653 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 01:28:29.093473 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:29.093444 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 04 01:28:29.093613 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:29.093493 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" podUID="d91e372a-c89d-41f6-a700-8c41a419a293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 04 01:28:30.052569 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:30.052529 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 01:28:30.052777 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:30.052593 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 01:28:30.093985 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:30.093953 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 04 01:28:30.094111 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:30.093999 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" podUID="d91e372a-c89d-41f6-a700-8c41a419a293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 04 01:28:31.052546 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:31.052516 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 01:28:31.052699 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:31.052573 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 01:28:31.093553 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:31.093525 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:28:31.093986 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:31.093829 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 04 01:28:31.093986 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:31.093878 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" podUID="d91e372a-c89d-41f6-a700-8c41a419a293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 04 01:28:32.052753 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:32.052710 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 01:28:32.052991 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:32.052768 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 01:28:32.052991 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:32.052814 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:32.053362 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:32.053337 7059 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="istio-proxy" containerStatusID={"Type":"cri-o","ID":"9ca62b5f6004c1fd875d6f86a094467c63563609f5130cad39e3064338b9346f"} pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" containerMessage="Container istio-proxy failed startup probe, will be restarted" Nov 04 01:28:32.053458 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:32.053395 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" containerID="cri-o://9ca62b5f6004c1fd875d6f86a094467c63563609f5130cad39e3064338b9346f" gracePeriod=30 Nov 04 01:28:32.093460 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:32.093432 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 04 01:28:32.093558 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:32.093474 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" podUID="d91e372a-c89d-41f6-a700-8c41a419a293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 04 01:28:33.093699 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:33.093652 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 04 01:28:33.094148 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:33.093714 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" podUID="d91e372a-c89d-41f6-a700-8c41a419a293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 04 01:28:34.098564 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:34.098522 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 04 01:28:34.099162 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:34.098598 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" podUID="d91e372a-c89d-41f6-a700-8c41a419a293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 04 01:28:35.098639 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:35.098613 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:28:35.099614 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:35.099585 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:28:36.497471 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:28:36.497432 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:28:37.168401 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:37.168363 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:28:37.168481 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:37.168448 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:28:37.168518 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:37.168479 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:28:37.573815 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:37.573777 7059 generic.go:356] "Generic (PLEG): container finished" podID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerID="9ca62b5f6004c1fd875d6f86a094467c63563609f5130cad39e3064338b9346f" exitCode=0 Nov 04 01:28:37.574323 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:37.573864 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" event={"ID":"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee","Type":"ContainerDied","Data":"9ca62b5f6004c1fd875d6f86a094467c63563609f5130cad39e3064338b9346f"} Nov 04 01:28:37.574323 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:37.574010 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" event={"ID":"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee","Type":"ContainerStarted","Data":"f5a574783f45086ddf36f57f3fb0e14bffa14f1ea7a350ab6836b1a563fa2adb"} Nov 04 01:28:38.052368 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:38.052335 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:38.052587 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:38.052499 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 04 01:28:38.052587 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:38.052558 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 04 01:28:39.057218 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:39.057188 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:39.597388 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:39.597359 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:39.598367 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:39.598344 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:28:52.347554 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:52.347447 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt"] Nov 04 01:28:52.355399 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:52.355368 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt" Nov 04 01:28:52.363800 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:52.363772 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-98f6f6-serving-cert\"" Nov 04 01:28:52.364019 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:52.363013 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt"] Nov 04 01:28:52.483855 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:52.481868 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb77296-8f8c-4ed8-8d70-f6cbdb09b888-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt\" (UID: \"3eb77296-8f8c-4ed8-8d70-f6cbdb09b888\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt" Nov 04 01:28:52.483855 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:52.481990 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c6xt\" (UniqueName: \"kubernetes.io/projected/3eb77296-8f8c-4ed8-8d70-f6cbdb09b888-kube-api-access-7c6xt\") pod \"maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt\" (UID: \"3eb77296-8f8c-4ed8-8d70-f6cbdb09b888\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt" Nov 04 01:28:52.483855 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:52.482190 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3eb77296-8f8c-4ed8-8d70-f6cbdb09b888-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt\" (UID: \"3eb77296-8f8c-4ed8-8d70-f6cbdb09b888\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt" Nov 04 01:28:52.483855 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:52.482323 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eb77296-8f8c-4ed8-8d70-f6cbdb09b888-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt\" (UID: \"3eb77296-8f8c-4ed8-8d70-f6cbdb09b888\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt" Nov 04 01:28:52.584530 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:52.584488 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb77296-8f8c-4ed8-8d70-f6cbdb09b888-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt\" (UID: \"3eb77296-8f8c-4ed8-8d70-f6cbdb09b888\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt" Nov 04 01:28:52.584722 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:52.584598 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c6xt\" (UniqueName: \"kubernetes.io/projected/3eb77296-8f8c-4ed8-8d70-f6cbdb09b888-kube-api-access-7c6xt\") pod \"maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt\" (UID: \"3eb77296-8f8c-4ed8-8d70-f6cbdb09b888\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt" Nov 04 01:28:52.584791 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:52.584749 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3eb77296-8f8c-4ed8-8d70-f6cbdb09b888-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt\" (UID: \"3eb77296-8f8c-4ed8-8d70-f6cbdb09b888\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt" Nov 04 01:28:52.584862 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:52.584848 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eb77296-8f8c-4ed8-8d70-f6cbdb09b888-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt\" (UID: \"3eb77296-8f8c-4ed8-8d70-f6cbdb09b888\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt" Nov 04 01:28:52.584939 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:28:52.584907 7059 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-98f6f6-serving-cert: secret "maas-api-e2e-shared-b-w3-98f6f6-serving-cert" not found Nov 04 01:28:52.585068 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:28:52.585013 7059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eb77296-8f8c-4ed8-8d70-f6cbdb09b888-maas-api-tls podName:3eb77296-8f8c-4ed8-8d70-f6cbdb09b888 nodeName:}" failed. No retries permitted until 2026-11-04 01:28:53.084990502 +0000 UTC m=+1818.026839400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3eb77296-8f8c-4ed8-8d70-f6cbdb09b888-maas-api-tls") pod "maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt" (UID: "3eb77296-8f8c-4ed8-8d70-f6cbdb09b888") : secret "maas-api-e2e-shared-b-w3-98f6f6-serving-cert" not found Nov 04 01:28:52.585331 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:52.585308 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb77296-8f8c-4ed8-8d70-f6cbdb09b888-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt\" (UID: \"3eb77296-8f8c-4ed8-8d70-f6cbdb09b888\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt" Nov 04 01:28:52.587726 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:52.587703 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eb77296-8f8c-4ed8-8d70-f6cbdb09b888-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt\" (UID: \"3eb77296-8f8c-4ed8-8d70-f6cbdb09b888\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt" Nov 04 01:28:52.600517 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:52.600460 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c6xt\" (UniqueName: \"kubernetes.io/projected/3eb77296-8f8c-4ed8-8d70-f6cbdb09b888-kube-api-access-7c6xt\") pod \"maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt\" (UID: \"3eb77296-8f8c-4ed8-8d70-f6cbdb09b888\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt" Nov 04 01:28:53.104093 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:53.104041 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3eb77296-8f8c-4ed8-8d70-f6cbdb09b888-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt\" (UID: \"3eb77296-8f8c-4ed8-8d70-f6cbdb09b888\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt" Nov 04 01:28:53.110993 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:53.109742 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3eb77296-8f8c-4ed8-8d70-f6cbdb09b888-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt\" (UID: \"3eb77296-8f8c-4ed8-8d70-f6cbdb09b888\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt" Nov 04 01:28:53.276263 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:53.276216 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt" Nov 04 01:28:53.685947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:53.678433 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt"] Nov 04 01:28:53.707204 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:53.698477 7059 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:28:53.817199 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:53.817166 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt" event={"ID":"3eb77296-8f8c-4ed8-8d70-f6cbdb09b888","Type":"ContainerStarted","Data":"df117a9947931518924d2d3b7c381998352243c2dbd676c2ca72f4686106d274"} Nov 04 01:28:54.862946 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:54.862133 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt" event={"ID":"3eb77296-8f8c-4ed8-8d70-f6cbdb09b888","Type":"ContainerStarted","Data":"415a634ba090cdacd3ab4109f467fc070bd88beb17ea444ee4a22210cee09d43"} Nov 04 01:28:54.862946 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:54.862249 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt" Nov 04 01:28:54.895305 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:28:54.895256 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt" podStartSLOduration=2.471862877 podStartE2EDuration="2.89524127s" totalImagesPullingTime="423.378393ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:54.889151015 +0000 UTC m=+1819.830999920" watchObservedRunningTime="2026-11-04 01:28:54.89524127 +0000 UTC m=+1819.837090173" Nov 04 01:29:00.880759 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:00.880725 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt" Nov 04 01:29:01.841760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:01.841723 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7"] Nov 04 01:29:01.851299 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:01.851267 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7" Nov 04 01:29:01.855037 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:01.855007 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7"] Nov 04 01:29:01.863474 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:01.863446 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-df6da4c0-serving-cert\"" Nov 04 01:29:01.901596 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:01.901565 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6hpt\" (UniqueName: \"kubernetes.io/projected/6203361d-87ea-4390-ac32-dfe81c7afd2a-kube-api-access-p6hpt\") pod \"maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7\" (UID: \"6203361d-87ea-4390-ac32-dfe81c7afd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7" Nov 04 01:29:01.902085 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:01.901642 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6203361d-87ea-4390-ac32-dfe81c7afd2a-maas-api-tls\") pod \"maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7\" (UID: \"6203361d-87ea-4390-ac32-dfe81c7afd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7" Nov 04 01:29:01.902085 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:01.901831 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6203361d-87ea-4390-ac32-dfe81c7afd2a-metrics-certs\") pod \"maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7\" (UID: \"6203361d-87ea-4390-ac32-dfe81c7afd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7" Nov 04 01:29:01.902085 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:01.902018 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6203361d-87ea-4390-ac32-dfe81c7afd2a-openshift-service-ca\") pod \"maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7\" (UID: \"6203361d-87ea-4390-ac32-dfe81c7afd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7" Nov 04 01:29:02.011839 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:02.008678 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6203361d-87ea-4390-ac32-dfe81c7afd2a-openshift-service-ca\") pod \"maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7\" (UID: \"6203361d-87ea-4390-ac32-dfe81c7afd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7" Nov 04 01:29:02.011839 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:02.009064 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6hpt\" (UniqueName: \"kubernetes.io/projected/6203361d-87ea-4390-ac32-dfe81c7afd2a-kube-api-access-p6hpt\") pod \"maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7\" (UID: \"6203361d-87ea-4390-ac32-dfe81c7afd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7" Nov 04 01:29:02.011839 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:02.009207 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6203361d-87ea-4390-ac32-dfe81c7afd2a-maas-api-tls\") pod \"maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7\" (UID: \"6203361d-87ea-4390-ac32-dfe81c7afd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7" Nov 04 01:29:02.011839 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:02.009345 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6203361d-87ea-4390-ac32-dfe81c7afd2a-metrics-certs\") pod \"maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7\" (UID: \"6203361d-87ea-4390-ac32-dfe81c7afd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7" Nov 04 01:29:02.017339 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:02.017045 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6203361d-87ea-4390-ac32-dfe81c7afd2a-openshift-service-ca\") pod \"maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7\" (UID: \"6203361d-87ea-4390-ac32-dfe81c7afd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7" Nov 04 01:29:02.018636 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:02.018609 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6203361d-87ea-4390-ac32-dfe81c7afd2a-maas-api-tls\") pod \"maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7\" (UID: \"6203361d-87ea-4390-ac32-dfe81c7afd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7" Nov 04 01:29:02.019636 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:02.019604 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6203361d-87ea-4390-ac32-dfe81c7afd2a-metrics-certs\") pod \"maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7\" (UID: \"6203361d-87ea-4390-ac32-dfe81c7afd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7" Nov 04 01:29:02.027059 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:02.027033 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6hpt\" (UniqueName: \"kubernetes.io/projected/6203361d-87ea-4390-ac32-dfe81c7afd2a-kube-api-access-p6hpt\") pod \"maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7\" (UID: \"6203361d-87ea-4390-ac32-dfe81c7afd2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7" Nov 04 01:29:02.185186 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:02.185143 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7" Nov 04 01:29:02.498945 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:29:02.494327 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6203361d_87ea_4390_ac32_dfe81c7afd2a.slice/crio-9ee73ce6f75cf84fa8bc4a3f039d2d22716273c0c5c1946579ad12655e86dc55 WatchSource:0}: Error finding container 9ee73ce6f75cf84fa8bc4a3f039d2d22716273c0c5c1946579ad12655e86dc55: Status 404 returned error can't find the container with id 9ee73ce6f75cf84fa8bc4a3f039d2d22716273c0c5c1946579ad12655e86dc55 Nov 04 01:29:02.498945 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:02.495810 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7"] Nov 04 01:29:02.808782 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:02.808707 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc"] Nov 04 01:29:02.815729 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:02.815700 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc" Nov 04 01:29:02.832984 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:02.832951 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10669ac8-e14a-4870-b1b5-f7331793d7a4-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc\" (UID: \"10669ac8-e14a-4870-b1b5-f7331793d7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc" Nov 04 01:29:02.833121 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:02.833014 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10669ac8-e14a-4870-b1b5-f7331793d7a4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc\" (UID: \"10669ac8-e14a-4870-b1b5-f7331793d7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc" Nov 04 01:29:02.833231 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:02.833207 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10669ac8-e14a-4870-b1b5-f7331793d7a4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc\" (UID: \"10669ac8-e14a-4870-b1b5-f7331793d7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc" Nov 04 01:29:02.833430 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:02.833409 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wp6w\" (UniqueName: \"kubernetes.io/projected/10669ac8-e14a-4870-b1b5-f7331793d7a4-kube-api-access-9wp6w\") pod \"maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc\" (UID: \"10669ac8-e14a-4870-b1b5-f7331793d7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc" Nov 04 01:29:02.851294 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:02.848669 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc"] Nov 04 01:29:02.851294 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:02.849117 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-75f7ae-serving-cert\"" Nov 04 01:29:02.935293 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:02.935253 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10669ac8-e14a-4870-b1b5-f7331793d7a4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc\" (UID: \"10669ac8-e14a-4870-b1b5-f7331793d7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc" Nov 04 01:29:02.935759 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:02.935423 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wp6w\" (UniqueName: \"kubernetes.io/projected/10669ac8-e14a-4870-b1b5-f7331793d7a4-kube-api-access-9wp6w\") pod \"maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc\" (UID: \"10669ac8-e14a-4870-b1b5-f7331793d7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc" Nov 04 01:29:02.935759 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:02.935588 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10669ac8-e14a-4870-b1b5-f7331793d7a4-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc\" (UID: \"10669ac8-e14a-4870-b1b5-f7331793d7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc" Nov 04 01:29:02.935759 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:02.935635 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10669ac8-e14a-4870-b1b5-f7331793d7a4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc\" (UID: \"10669ac8-e14a-4870-b1b5-f7331793d7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc" Nov 04 01:29:02.936568 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:02.936541 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10669ac8-e14a-4870-b1b5-f7331793d7a4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc\" (UID: \"10669ac8-e14a-4870-b1b5-f7331793d7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc" Nov 04 01:29:02.938198 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:02.938174 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10669ac8-e14a-4870-b1b5-f7331793d7a4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc\" (UID: \"10669ac8-e14a-4870-b1b5-f7331793d7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc" Nov 04 01:29:02.938383 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:02.938362 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10669ac8-e14a-4870-b1b5-f7331793d7a4-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc\" (UID: \"10669ac8-e14a-4870-b1b5-f7331793d7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc" Nov 04 01:29:02.944114 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:02.943897 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wp6w\" (UniqueName: \"kubernetes.io/projected/10669ac8-e14a-4870-b1b5-f7331793d7a4-kube-api-access-9wp6w\") pod \"maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc\" (UID: \"10669ac8-e14a-4870-b1b5-f7331793d7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc" Nov 04 01:29:02.944206 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:02.944169 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7" event={"ID":"6203361d-87ea-4390-ac32-dfe81c7afd2a","Type":"ContainerStarted","Data":"9ee73ce6f75cf84fa8bc4a3f039d2d22716273c0c5c1946579ad12655e86dc55"} Nov 04 01:29:03.183958 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:03.173102 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc" Nov 04 01:29:03.598529 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:03.598425 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc"] Nov 04 01:29:03.993050 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:03.993007 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc" event={"ID":"10669ac8-e14a-4870-b1b5-f7331793d7a4","Type":"ContainerStarted","Data":"90473e4e5c24e1c8c0f504d432e2d5f5640aa747c864497c33de0bfc1c355330"} Nov 04 01:29:04.000853 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:04.000813 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7" event={"ID":"6203361d-87ea-4390-ac32-dfe81c7afd2a","Type":"ContainerStarted","Data":"02ecf1945f9473554fd38bb08601da601797db2c95cfdbcbe307503514ec950e"} Nov 04 01:29:04.016462 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:04.016430 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7" Nov 04 01:29:04.070949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:04.062157 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7" podStartSLOduration=2.5281436470000003 podStartE2EDuration="3.062137155s" totalImagesPullingTime="533.993508ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:04.041442547 +0000 UTC m=+1828.983291451" watchObservedRunningTime="2026-11-04 01:29:04.062137155 +0000 UTC m=+1829.003986059" Nov 04 01:29:05.011459 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:05.011425 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc" event={"ID":"10669ac8-e14a-4870-b1b5-f7331793d7a4","Type":"ContainerStarted","Data":"b8d9137e8aa438da6ff8d08c1cae0115d5203246e541c2d9c6074c9179ec5c12"} Nov 04 01:29:05.011990 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:05.011543 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc" Nov 04 01:29:05.039469 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:05.039417 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc" podStartSLOduration=2.474066274 podStartE2EDuration="3.039404817s" totalImagesPullingTime="565.338543ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:05.037128514 +0000 UTC m=+1829.978977418" watchObservedRunningTime="2026-11-04 01:29:05.039404817 +0000 UTC m=+1829.981253738" Nov 04 01:29:06.531675 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:29:06.531633 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:29:11.025332 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:11.025303 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc" Nov 04 01:29:11.025810 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:11.025767 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7" Nov 04 01:29:26.300100 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:26.300034 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq"] Nov 04 01:29:26.319989 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:26.319954 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp"] Nov 04 01:29:36.499889 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:29:36.499846 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:29:39.953782 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:39.953743 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt"] Nov 04 01:29:39.961200 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:39.961175 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt" Nov 04 01:29:39.980155 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:39.980123 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt"] Nov 04 01:29:39.985198 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:39.985172 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-5f572a-serving-cert\"" Nov 04 01:29:40.048894 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:40.048856 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd37362b-7164-4401-bd4b-08af565a8bc4-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt\" (UID: \"cd37362b-7164-4401-bd4b-08af565a8bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt" Nov 04 01:29:40.049085 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:40.048955 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd37362b-7164-4401-bd4b-08af565a8bc4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt\" (UID: \"cd37362b-7164-4401-bd4b-08af565a8bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt" Nov 04 01:29:40.049146 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:40.049124 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5n4r\" (UniqueName: \"kubernetes.io/projected/cd37362b-7164-4401-bd4b-08af565a8bc4-kube-api-access-w5n4r\") pod \"maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt\" (UID: \"cd37362b-7164-4401-bd4b-08af565a8bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt" Nov 04 01:29:40.049356 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:40.049337 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd37362b-7164-4401-bd4b-08af565a8bc4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt\" (UID: \"cd37362b-7164-4401-bd4b-08af565a8bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt" Nov 04 01:29:40.150664 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:40.150626 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5n4r\" (UniqueName: \"kubernetes.io/projected/cd37362b-7164-4401-bd4b-08af565a8bc4-kube-api-access-w5n4r\") pod \"maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt\" (UID: \"cd37362b-7164-4401-bd4b-08af565a8bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt" Nov 04 01:29:40.150872 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:40.150857 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd37362b-7164-4401-bd4b-08af565a8bc4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt\" (UID: \"cd37362b-7164-4401-bd4b-08af565a8bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt" Nov 04 01:29:40.151083 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:40.151066 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd37362b-7164-4401-bd4b-08af565a8bc4-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt\" (UID: \"cd37362b-7164-4401-bd4b-08af565a8bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt" Nov 04 01:29:40.151216 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:40.151097 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd37362b-7164-4401-bd4b-08af565a8bc4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt\" (UID: \"cd37362b-7164-4401-bd4b-08af565a8bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt" Nov 04 01:29:40.151653 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:40.151570 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd37362b-7164-4401-bd4b-08af565a8bc4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt\" (UID: \"cd37362b-7164-4401-bd4b-08af565a8bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt" Nov 04 01:29:40.153822 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:40.153798 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd37362b-7164-4401-bd4b-08af565a8bc4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt\" (UID: \"cd37362b-7164-4401-bd4b-08af565a8bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt" Nov 04 01:29:40.153903 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:40.153819 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd37362b-7164-4401-bd4b-08af565a8bc4-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt\" (UID: \"cd37362b-7164-4401-bd4b-08af565a8bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt" Nov 04 01:29:40.163131 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:40.163105 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5n4r\" (UniqueName: \"kubernetes.io/projected/cd37362b-7164-4401-bd4b-08af565a8bc4-kube-api-access-w5n4r\") pod \"maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt\" (UID: \"cd37362b-7164-4401-bd4b-08af565a8bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt" Nov 04 01:29:40.285490 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:40.285404 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt" Nov 04 01:29:40.587461 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:29:40.587421 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd37362b_7164_4401_bd4b_08af565a8bc4.slice/crio-bc56e19d05e9d00125aafe85c3cccc71cc4cbae405048255780325aeac8847ae WatchSource:0}: Error finding container bc56e19d05e9d00125aafe85c3cccc71cc4cbae405048255780325aeac8847ae: Status 404 returned error can't find the container with id bc56e19d05e9d00125aafe85c3cccc71cc4cbae405048255780325aeac8847ae Nov 04 01:29:40.589143 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:40.589114 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt"] Nov 04 01:29:41.332051 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.330608 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" containerID="cri-o://c37241846aafa2ba33d0da06870d86efc817f546bee6d0e2513e0fa6d879e77b" gracePeriod=45 Nov 04 01:29:41.349636 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.349532 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" containerID="cri-o://05600c56a88be7feb214cc76570f270fba59e1ef37abdd77c664a3c20a413817" gracePeriod=45 Nov 04 01:29:41.471536 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.471473 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": dial tcp 10.133.0.75:8000: connect: connection refused" Nov 04 01:29:41.608482 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.608441 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt" event={"ID":"cd37362b-7164-4401-bd4b-08af565a8bc4","Type":"ContainerStarted","Data":"725f457b7eb25f2c353d7a183873b78ce64565ffb92bcb58afacd4564d163a65"} Nov 04 01:29:41.608598 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.608517 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt" event={"ID":"cd37362b-7164-4401-bd4b-08af565a8bc4","Type":"ContainerStarted","Data":"bc56e19d05e9d00125aafe85c3cccc71cc4cbae405048255780325aeac8847ae"} Nov 04 01:29:41.609868 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.609784 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt" Nov 04 01:29:41.614418 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.614374 7059 generic.go:356] "Generic (PLEG): container finished" podID="64d89f45-696f-4871-8178-566852da13f4" containerID="c37241846aafa2ba33d0da06870d86efc817f546bee6d0e2513e0fa6d879e77b" exitCode=0 Nov 04 01:29:41.614540 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.614510 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" event={"ID":"64d89f45-696f-4871-8178-566852da13f4","Type":"ContainerDied","Data":"c37241846aafa2ba33d0da06870d86efc817f546bee6d0e2513e0fa6d879e77b"} Nov 04 01:29:41.616841 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.616814 7059 generic.go:356] "Generic (PLEG): container finished" podID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerID="05600c56a88be7feb214cc76570f270fba59e1ef37abdd77c664a3c20a413817" exitCode=0 Nov 04 01:29:41.617025 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.616994 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" event={"ID":"8b3ccd4d-549f-4621-908f-5be7a1e2a99a","Type":"ContainerDied","Data":"05600c56a88be7feb214cc76570f270fba59e1ef37abdd77c664a3c20a413817"} Nov 04 01:29:41.644382 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.642911 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt" podStartSLOduration=2.138431562 podStartE2EDuration="2.642892363s" totalImagesPullingTime="504.460801ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:41.635252674 +0000 UTC m=+1866.577101578" watchObservedRunningTime="2026-11-04 01:29:41.642892363 +0000 UTC m=+1866.584741268" Nov 04 01:29:41.725582 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.725557 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:29:41.730193 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.730163 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:29:41.740610 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.740588 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64d89f45-696f-4871-8178-566852da13f4-tls-certs\" (UniqueName: \"kubernetes.io/secret/64d89f45-696f-4871-8178-566852da13f4-tls-certs\") pod \"64d89f45-696f-4871-8178-566852da13f4\" (UID: \"64d89f45-696f-4871-8178-566852da13f4\") " Nov 04 01:29:41.740791 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.740778 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64d89f45-696f-4871-8178-566852da13f4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/64d89f45-696f-4871-8178-566852da13f4-dshm\") pod \"64d89f45-696f-4871-8178-566852da13f4\" (UID: \"64d89f45-696f-4871-8178-566852da13f4\") " Nov 04 01:29:41.740868 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.740799 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64d89f45-696f-4871-8178-566852da13f4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64d89f45-696f-4871-8178-566852da13f4-model-cache\") pod \"64d89f45-696f-4871-8178-566852da13f4\" (UID: \"64d89f45-696f-4871-8178-566852da13f4\") " Nov 04 01:29:41.740970 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.740936 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64d89f45-696f-4871-8178-566852da13f4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64d89f45-696f-4871-8178-566852da13f4-tmp-dir\") pod \"64d89f45-696f-4871-8178-566852da13f4\" (UID: \"64d89f45-696f-4871-8178-566852da13f4\") " Nov 04 01:29:41.741029 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.740989 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64d89f45-696f-4871-8178-566852da13f4-home\" (UniqueName: \"kubernetes.io/empty-dir/64d89f45-696f-4871-8178-566852da13f4-home\") pod \"64d89f45-696f-4871-8178-566852da13f4\" (UID: \"64d89f45-696f-4871-8178-566852da13f4\") " Nov 04 01:29:41.741029 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.741023 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64d89f45-696f-4871-8178-566852da13f4-kube-api-access-t8vsp\" (UniqueName: \"kubernetes.io/projected/64d89f45-696f-4871-8178-566852da13f4-kube-api-access-t8vsp\") pod \"64d89f45-696f-4871-8178-566852da13f4\" (UID: \"64d89f45-696f-4871-8178-566852da13f4\") " Nov 04 01:29:41.741297 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.741267 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d89f45-696f-4871-8178-566852da13f4-model-cache" pod "64d89f45-696f-4871-8178-566852da13f4" (UID: "64d89f45-696f-4871-8178-566852da13f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:41.741474 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.741339 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d89f45-696f-4871-8178-566852da13f4-home" pod "64d89f45-696f-4871-8178-566852da13f4" (UID: "64d89f45-696f-4871-8178-566852da13f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:41.741474 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.741388 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d89f45-696f-4871-8178-566852da13f4-tmp-dir" pod "64d89f45-696f-4871-8178-566852da13f4" (UID: "64d89f45-696f-4871-8178-566852da13f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:41.741800 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.741776 7059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64d89f45-696f-4871-8178-566852da13f4-model-cache\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:41.741960 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.741803 7059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64d89f45-696f-4871-8178-566852da13f4-tmp-dir\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:41.741960 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.741817 7059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64d89f45-696f-4871-8178-566852da13f4-home\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:41.745006 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.743699 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64d89f45-696f-4871-8178-566852da13f4-kube-api-access-t8vsp" pod "64d89f45-696f-4871-8178-566852da13f4" (UID: "64d89f45-696f-4871-8178-566852da13f4"). InnerVolumeSpecName "kube-api-access-t8vsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:29:41.771717 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.747384 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d89f45-696f-4871-8178-566852da13f4-dshm" pod "64d89f45-696f-4871-8178-566852da13f4" (UID: "64d89f45-696f-4871-8178-566852da13f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:41.771717 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.747443 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64d89f45-696f-4871-8178-566852da13f4-tls-certs" pod "64d89f45-696f-4871-8178-566852da13f4" (UID: "64d89f45-696f-4871-8178-566852da13f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:29:41.844611 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.844573 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-tmp-dir\") pod \"8b3ccd4d-549f-4621-908f-5be7a1e2a99a\" (UID: \"8b3ccd4d-549f-4621-908f-5be7a1e2a99a\") " Nov 04 01:29:41.844801 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.844658 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-dshm\") pod \"8b3ccd4d-549f-4621-908f-5be7a1e2a99a\" (UID: \"8b3ccd4d-549f-4621-908f-5be7a1e2a99a\") " Nov 04 01:29:41.844801 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.844756 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-kube-api-access-9xf5m\" (UniqueName: \"kubernetes.io/projected/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-kube-api-access-9xf5m\") pod \"8b3ccd4d-549f-4621-908f-5be7a1e2a99a\" (UID: \"8b3ccd4d-549f-4621-908f-5be7a1e2a99a\") " Nov 04 01:29:41.844957 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.844840 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-home\" (UniqueName: \"kubernetes.io/empty-dir/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-home\") pod \"8b3ccd4d-549f-4621-908f-5be7a1e2a99a\" (UID: \"8b3ccd4d-549f-4621-908f-5be7a1e2a99a\") " Nov 04 01:29:41.845032 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.845011 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-tls-certs\" (UniqueName: \"kubernetes.io/secret/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-tls-certs\") pod \"8b3ccd4d-549f-4621-908f-5be7a1e2a99a\" (UID: \"8b3ccd4d-549f-4621-908f-5be7a1e2a99a\") " Nov 04 01:29:41.845093 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.845061 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-model-cache\") pod \"8b3ccd4d-549f-4621-908f-5be7a1e2a99a\" (UID: \"8b3ccd4d-549f-4621-908f-5be7a1e2a99a\") " Nov 04 01:29:41.845980 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.845894 7059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64d89f45-696f-4871-8178-566852da13f4-dshm\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:41.845980 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.845948 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8vsp\" (UniqueName: \"kubernetes.io/projected/64d89f45-696f-4871-8178-566852da13f4-kube-api-access-t8vsp\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:41.845980 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.845964 7059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64d89f45-696f-4871-8178-566852da13f4-tls-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:41.846540 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.846515 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-model-cache" pod "8b3ccd4d-549f-4621-908f-5be7a1e2a99a" (UID: "8b3ccd4d-549f-4621-908f-5be7a1e2a99a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:41.846774 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.846740 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-home" pod "8b3ccd4d-549f-4621-908f-5be7a1e2a99a" (UID: "8b3ccd4d-549f-4621-908f-5be7a1e2a99a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:41.847323 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.847292 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-tmp-dir" pod "8b3ccd4d-549f-4621-908f-5be7a1e2a99a" (UID: "8b3ccd4d-549f-4621-908f-5be7a1e2a99a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:41.860669 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.849385 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-tls-certs" pod "8b3ccd4d-549f-4621-908f-5be7a1e2a99a" (UID: "8b3ccd4d-549f-4621-908f-5be7a1e2a99a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:29:41.860669 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.857173 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-dshm" pod "8b3ccd4d-549f-4621-908f-5be7a1e2a99a" (UID: "8b3ccd4d-549f-4621-908f-5be7a1e2a99a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:41.863314 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.863292 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-kube-api-access-9xf5m" pod "8b3ccd4d-549f-4621-908f-5be7a1e2a99a" (UID: "8b3ccd4d-549f-4621-908f-5be7a1e2a99a"). InnerVolumeSpecName "kube-api-access-9xf5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:29:41.947437 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.947404 7059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-model-cache\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:41.947437 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.947435 7059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-tmp-dir\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:41.947437 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.947444 7059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-dshm\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:41.947655 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.947454 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xf5m\" (UniqueName: \"kubernetes.io/projected/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-kube-api-access-9xf5m\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:41.947655 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.947465 7059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-home\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:41.947655 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:41.947473 7059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b3ccd4d-549f-4621-908f-5be7a1e2a99a-tls-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:42.573073 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:42.572987 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:42.628205 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:42.628177 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" Nov 04 01:29:42.628355 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:42.628221 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq" event={"ID":"64d89f45-696f-4871-8178-566852da13f4","Type":"ContainerDied","Data":"1edde9ed24f4290fc6db75ef21a9e2c3b733ce6fc6780eeb100e826597971532"} Nov 04 01:29:42.628355 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:42.628285 7059 scope.go:122] "RemoveContainer" containerID="c37241846aafa2ba33d0da06870d86efc817f546bee6d0e2513e0fa6d879e77b" Nov 04 01:29:42.630039 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:42.630019 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" Nov 04 01:29:42.630244 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:42.630055 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp" event={"ID":"8b3ccd4d-549f-4621-908f-5be7a1e2a99a","Type":"ContainerDied","Data":"7a49bcdc0eb41aa449b2559b8f6266bea7504981100fab0201a54ceb6651596c"} Nov 04 01:29:42.642367 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:42.642347 7059 scope.go:122] "RemoveContainer" containerID="05600c56a88be7feb214cc76570f270fba59e1ef37abdd77c664a3c20a413817" Nov 04 01:29:42.662279 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:42.662251 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp"] Nov 04 01:29:42.666862 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:42.666831 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ee876a/premium-w0-ee876a-kserve-67b45d4948-n6tcp"] Nov 04 01:29:42.678031 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:42.678009 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq"] Nov 04 01:29:42.684086 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:42.684066 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ee876a/distinct-2-w0-ee876a-kserve-66ffc69696-hk7rq"] Nov 04 01:29:43.186975 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:43.185027 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64d89f45-696f-4871-8178-566852da13f4" path="/var/lib/kubelet/pods/64d89f45-696f-4871-8178-566852da13f4/volumes" Nov 04 01:29:43.186975 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:43.185726 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" path="/var/lib/kubelet/pods/8b3ccd4d-549f-4621-908f-5be7a1e2a99a/volumes" Nov 04 01:29:43.823203 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:43.823164 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ee876a-54f4fd798c-djxkh"] Nov 04 01:29:46.845186 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:46.845147 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9"] Nov 04 01:29:46.846467 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:46.846440 7059 cpu_manager.go:414] "containerMap: removing container" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" Nov 04 01:29:46.846467 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:46.846468 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" Nov 04 01:29:46.846618 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:46.846560 7059 cpu_manager.go:414] "containerMap: removing container" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" Nov 04 01:29:46.846618 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:46.846569 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" Nov 04 01:29:46.846797 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:46.846781 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b3ccd4d-549f-4621-908f-5be7a1e2a99a" containerName="main" Nov 04 01:29:46.846841 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:46.846820 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="64d89f45-696f-4871-8178-566852da13f4" containerName="main" Nov 04 01:29:46.879915 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:46.879881 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9"] Nov 04 01:29:46.880131 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:46.880031 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:29:46.938038 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:46.938002 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-df6da4c0-openshift-default-dockercfg-78n5r\"" Nov 04 01:29:46.952577 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:46.952545 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-workload-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:29:46.952775 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:46.952754 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istio-token\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:29:46.952849 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:46.952802 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istio-podinfo\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:29:46.952898 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:46.952872 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istiod-ca-cert\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:29:46.952970 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:46.952898 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t949\" (UniqueName: \"kubernetes.io/projected/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-kube-api-access-8t949\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:29:46.953021 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:46.953008 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-workload-certs\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:29:46.953094 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:46.953082 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istio-envoy\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:29:46.953299 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:46.953275 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istio-data\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:29:46.953403 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:46.953382 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-credential-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:29:47.055461 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:47.055427 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-workload-certs\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:29:47.055590 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:47.055515 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istio-envoy\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:29:47.055667 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:47.055642 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istio-data\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:29:47.055728 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:47.055699 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-credential-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:29:47.055805 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:47.055789 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-workload-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:29:47.055986 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:47.055853 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-workload-certs\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:29:47.056089 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:47.056038 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istio-token\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:29:47.056146 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:47.056094 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istio-podinfo\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:29:47.056146 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:47.056121 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istiod-ca-cert\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:29:47.056249 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:47.056152 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t949\" (UniqueName: \"kubernetes.io/projected/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-kube-api-access-8t949\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:29:47.056716 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:47.056691 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-workload-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:29:47.056877 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:47.056046 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istio-data\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:29:47.057306 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:47.057260 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istiod-ca-cert\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:29:47.057538 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:47.057516 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-credential-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:29:47.058548 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:47.058526 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istio-envoy\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:29:47.059821 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:47.059799 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istio-podinfo\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:29:47.073910 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:47.073892 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t949\" (UniqueName: \"kubernetes.io/projected/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-kube-api-access-8t949\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:29:47.074154 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:47.074136 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istio-token\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:29:47.201090 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:47.201061 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:29:47.522639 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:29:47.522609 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c935f3_7c45_456d_81a7_c2bd6a38e1a0.slice/crio-17de2a259fa153494a7c5846512de9aea3a05fc2134e3d1873a7ce6a438498bd WatchSource:0}: Error finding container 17de2a259fa153494a7c5846512de9aea3a05fc2134e3d1873a7ce6a438498bd: Status 404 returned error can't find the container with id 17de2a259fa153494a7c5846512de9aea3a05fc2134e3d1873a7ce6a438498bd Nov 04 01:29:47.523037 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:47.523012 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9"] Nov 04 01:29:47.526068 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:47.526026 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:29:47.526159 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:47.526117 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:29:47.526194 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:47.526158 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:29:47.695647 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:47.695601 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" event={"ID":"43c935f3-7c45-456d-81a7-c2bd6a38e1a0","Type":"ContainerStarted","Data":"54ecb7f98862fcee26e93182e392cebec2482f930fa85e996e9409be51813db9"} Nov 04 01:29:47.695834 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:47.695705 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" event={"ID":"43c935f3-7c45-456d-81a7-c2bd6a38e1a0","Type":"ContainerStarted","Data":"17de2a259fa153494a7c5846512de9aea3a05fc2134e3d1873a7ce6a438498bd"} Nov 04 01:29:47.804594 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:47.802561 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" podStartSLOduration=1.802541709 podStartE2EDuration="1.802541709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:47.793193823 +0000 UTC m=+1872.735042727" watchObservedRunningTime="2026-11-04 01:29:47.802541709 +0000 UTC m=+1872.744390618" Nov 04 01:29:48.201903 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:48.201873 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:29:48.203176 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:48.203149 7059 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:29:48.203296 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:48.203204 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" podUID="43c935f3-7c45-456d-81a7-c2bd6a38e1a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 04 01:29:48.668662 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:48.668579 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt" Nov 04 01:29:48.823451 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:48.823399 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee876a-54f4fd798c-djxkh" podUID="a70660e3-f657-4f28-9743-651390c8ae3f" containerName="payload-pre-processing" containerID="cri-o://1859918638a2b822c70df32a32e0fcc5e1ad34c9d4f31488d28724f7c051a4ed" gracePeriod=25 Nov 04 01:29:49.230986 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:49.227644 7059 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:29:49.230986 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:49.227705 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" podUID="43c935f3-7c45-456d-81a7-c2bd6a38e1a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 04 01:29:50.201581 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:50.201545 7059 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:29:50.201775 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:50.201610 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" podUID="43c935f3-7c45-456d-81a7-c2bd6a38e1a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 04 01:29:51.201605 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:51.201568 7059 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:29:51.202018 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:51.201622 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" podUID="43c935f3-7c45-456d-81a7-c2bd6a38e1a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 04 01:29:52.201592 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:52.201535 7059 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:29:52.201753 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:52.201608 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" podUID="43c935f3-7c45-456d-81a7-c2bd6a38e1a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 04 01:29:52.881274 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:52.879054 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb"] Nov 04 01:29:52.886165 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:52.886137 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:29:52.888733 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:52.888447 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb"] Nov 04 01:29:52.973542 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:52.972555 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-75f7ae-openshift-default-dockercfg-v4pjw\"" Nov 04 01:29:53.023720 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.023687 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-workload-certs\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:29:53.023916 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.023815 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-istio-envoy\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:29:53.023916 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.023868 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-workload-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:29:53.023916 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.023907 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-istio-data\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:29:53.024120 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.024089 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx2tr\" (UniqueName: \"kubernetes.io/projected/612b8bca-a31e-4126-a517-ff449f55acd8-kube-api-access-tx2tr\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:29:53.024189 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.024131 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-credential-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:29:53.024389 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.024363 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/612b8bca-a31e-4126-a517-ff449f55acd8-istio-token\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:29:53.024458 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.024432 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/612b8bca-a31e-4126-a517-ff449f55acd8-istiod-ca-cert\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:29:53.024563 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.024549 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/612b8bca-a31e-4126-a517-ff449f55acd8-istio-podinfo\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:29:53.125649 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.125616 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-istio-envoy\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:29:53.125813 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.125662 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-workload-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:29:53.125813 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.125686 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-istio-data\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:29:53.125889 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.125831 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx2tr\" (UniqueName: \"kubernetes.io/projected/612b8bca-a31e-4126-a517-ff449f55acd8-kube-api-access-tx2tr\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:29:53.125889 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.125878 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-credential-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:29:53.126149 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.126124 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-istio-data\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:29:53.126331 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.126141 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/612b8bca-a31e-4126-a517-ff449f55acd8-istio-token\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:29:53.126435 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.126415 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/612b8bca-a31e-4126-a517-ff449f55acd8-istiod-ca-cert\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:29:53.126577 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.126169 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-workload-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:29:53.126652 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.126523 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/612b8bca-a31e-4126-a517-ff449f55acd8-istio-podinfo\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:29:53.126722 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.126259 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-credential-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:29:53.126722 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.126687 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-workload-certs\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:29:53.127054 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.127037 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-workload-certs\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:29:53.127305 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.127281 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/612b8bca-a31e-4126-a517-ff449f55acd8-istiod-ca-cert\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:29:53.128312 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.128293 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-istio-envoy\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:29:53.129828 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.129811 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/612b8bca-a31e-4126-a517-ff449f55acd8-istio-podinfo\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:29:53.134267 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.134215 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx2tr\" (UniqueName: \"kubernetes.io/projected/612b8bca-a31e-4126-a517-ff449f55acd8-kube-api-access-tx2tr\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:29:53.134594 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.134574 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/612b8bca-a31e-4126-a517-ff449f55acd8-istio-token\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:29:53.201707 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.201674 7059 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:29:53.201843 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.201716 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" podUID="43c935f3-7c45-456d-81a7-c2bd6a38e1a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 04 01:29:53.211257 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.211233 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:29:53.571985 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.563082 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb"] Nov 04 01:29:53.585705 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.585671 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:29:53.585781 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.585740 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:29:53.585781 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.585773 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:29:53.810610 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.810080 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" event={"ID":"612b8bca-a31e-4126-a517-ff449f55acd8","Type":"ContainerStarted","Data":"ba4e37ce5194dddbbc75cbaeecb38504bd9e8b011e116861608b683a10b3604b"} Nov 04 01:29:53.810610 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.810141 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" event={"ID":"612b8bca-a31e-4126-a517-ff449f55acd8","Type":"ContainerStarted","Data":"48976f41dccc87c032e353d388819ea71d843994ea19e60183dcdc520c587549"} Nov 04 01:29:53.860946 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:53.858640 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" podStartSLOduration=1.858620717 podStartE2EDuration="1.858620717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:53.83185007 +0000 UTC m=+1878.773698974" watchObservedRunningTime="2026-11-04 01:29:53.858620717 +0000 UTC m=+1878.800469622" Nov 04 01:29:54.212269 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:54.212227 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:29:54.214565 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:54.214529 7059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:29:54.214731 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:54.214583 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" podUID="612b8bca-a31e-4126-a517-ff449f55acd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 04 01:29:54.220398 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:54.220354 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:29:54.222209 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:54.222177 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:29:54.224955 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:54.224579 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:29:55.212155 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:55.212042 7059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:29:55.212341 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:55.212186 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" podUID="612b8bca-a31e-4126-a517-ff449f55acd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 04 01:29:56.212404 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:56.212361 7059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:29:56.212896 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:56.212420 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" podUID="612b8bca-a31e-4126-a517-ff449f55acd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 04 01:29:57.212087 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:57.212056 7059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:29:57.212252 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:57.212107 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" podUID="612b8bca-a31e-4126-a517-ff449f55acd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 04 01:29:57.416778 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:57.416567 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee876a-54f4fd798c-djxkh" podUID="a70660e3-f657-4f28-9743-651390c8ae3f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:29:58.211810 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:58.211776 7059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:29:58.212039 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:58.211839 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" podUID="612b8bca-a31e-4126-a517-ff449f55acd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 04 01:29:58.470720 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:58.470635 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll"] Nov 04 01:29:58.471129 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:58.470842 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" podUID="86db204f-72c0-48a4-817d-34067fbf8f0a" containerName="istio-proxy" containerID="cri-o://ac60ec4fb6d14ad97c1911e6248151c036c2f4584abdb302a82d8ce69e8c9b4f" gracePeriod=30 Nov 04 01:29:59.212001 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:59.211970 7059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:29:59.212185 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:29:59.212022 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" podUID="612b8bca-a31e-4126-a517-ff449f55acd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 04 01:30:00.218796 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:00.218760 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:30:00.219326 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:00.219265 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:30:00.220208 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:00.220185 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:30:01.887982 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:01.867578 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz"] Nov 04 01:30:01.900338 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:01.900309 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:30:01.956308 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:01.950345 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ee575f-2616-424f-a7ea-79b03b554a5e-istio-token\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:30:01.956308 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:01.950399 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-istio-envoy\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:30:01.956308 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:01.950498 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdt66\" (UniqueName: \"kubernetes.io/projected/03ee575f-2616-424f-a7ea-79b03b554a5e-kube-api-access-cdt66\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:30:01.956308 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:01.950573 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-istio-data\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:30:01.956308 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:01.950636 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-workload-certs\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:30:01.956308 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:01.950671 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ee575f-2616-424f-a7ea-79b03b554a5e-istiod-ca-cert\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:30:01.956308 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:01.950808 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ee575f-2616-424f-a7ea-79b03b554a5e-istio-podinfo\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:30:01.956308 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:01.950838 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-workload-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:30:01.956308 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:01.950885 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-credential-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:30:01.981628 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:01.973008 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz"] Nov 04 01:30:01.981628 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:01.981543 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:30:01.981874 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:01.981793 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz"] Nov 04 01:30:01.992156 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:01.992114 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz"] Nov 04 01:30:02.081486 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.081437 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-istio-envoy\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:30:02.093061 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.092968 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ee575f-2616-424f-a7ea-79b03b554a5e-istiod-ca-cert\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:30:02.093061 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.093021 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvq7k\" (UniqueName: \"kubernetes.io/projected/50745197-72c7-41bf-a41e-23a796a4d4d0-kube-api-access-fvq7k\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:30:02.093061 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.093066 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50745197-72c7-41bf-a41e-23a796a4d4d0-istio-podinfo\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:30:02.093367 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.093193 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ee575f-2616-424f-a7ea-79b03b554a5e-istio-podinfo\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:30:02.093367 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.093219 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-workload-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:30:02.093367 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.093262 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-credential-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:30:02.093367 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.093296 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-workload-certs\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:30:02.093367 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.093359 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-workload-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:30:02.093614 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.093466 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ee575f-2616-424f-a7ea-79b03b554a5e-istio-token\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:30:02.093614 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.093498 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-istio-envoy\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:30:02.093614 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.093513 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-istio-data\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:30:02.093761 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.093613 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50745197-72c7-41bf-a41e-23a796a4d4d0-istio-token\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:30:02.093761 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.093639 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdt66\" (UniqueName: \"kubernetes.io/projected/03ee575f-2616-424f-a7ea-79b03b554a5e-kube-api-access-cdt66\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:30:02.093761 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.093694 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-credential-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:30:02.093761 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.093732 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-istio-data\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:30:02.093990 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.093763 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50745197-72c7-41bf-a41e-23a796a4d4d0-istiod-ca-cert\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:30:02.093990 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.093799 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-workload-certs\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:30:02.094365 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.094324 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-workload-certs\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:30:02.095343 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.095309 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-istio-data\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:30:02.095496 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.095362 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-credential-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:30:02.095602 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.095570 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-workload-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:30:02.096244 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.096222 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ee575f-2616-424f-a7ea-79b03b554a5e-istiod-ca-cert\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:30:02.098777 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.098581 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-istio-envoy\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:30:02.100424 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.100386 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ee575f-2616-424f-a7ea-79b03b554a5e-istio-podinfo\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:30:02.104671 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.104644 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ee575f-2616-424f-a7ea-79b03b554a5e-istio-token\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:30:02.104851 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.104826 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdt66\" (UniqueName: \"kubernetes.io/projected/03ee575f-2616-424f-a7ea-79b03b554a5e-kube-api-access-cdt66\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:30:02.201991 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.201949 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50745197-72c7-41bf-a41e-23a796a4d4d0-istio-token\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:30:02.202321 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.202300 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-credential-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:30:02.202522 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.202493 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50745197-72c7-41bf-a41e-23a796a4d4d0-istiod-ca-cert\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:30:02.202687 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.202669 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-istio-envoy\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:30:02.202809 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.202795 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvq7k\" (UniqueName: \"kubernetes.io/projected/50745197-72c7-41bf-a41e-23a796a4d4d0-kube-api-access-fvq7k\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:30:02.202938 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.202909 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50745197-72c7-41bf-a41e-23a796a4d4d0-istio-podinfo\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:30:02.203299 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.203281 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-workload-certs\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:30:02.203458 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.203443 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-workload-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:30:02.203716 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.203688 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-istio-data\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:30:02.203798 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.203740 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-credential-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:30:02.204094 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.204050 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-workload-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:30:02.204847 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.204588 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-istio-data\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:30:02.204847 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.204810 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50745197-72c7-41bf-a41e-23a796a4d4d0-istiod-ca-cert\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:30:02.206331 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.206307 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-workload-certs\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:30:02.207220 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.206353 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-istio-envoy\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:30:02.207426 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.207403 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50745197-72c7-41bf-a41e-23a796a4d4d0-istio-podinfo\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:30:02.214503 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.214420 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50745197-72c7-41bf-a41e-23a796a4d4d0-istio-token\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:30:02.216450 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.216422 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvq7k\" (UniqueName: \"kubernetes.io/projected/50745197-72c7-41bf-a41e-23a796a4d4d0-kube-api-access-fvq7k\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:30:02.280270 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.280227 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:30:02.325589 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.325551 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:30:02.609609 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:30:02.609194 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ee575f_2616_424f_a7ea_79b03b554a5e.slice/crio-25cd3bbafac12c9de5648e8837040960f9abf21b39874a8f3c4c37c5d1ec6b56 WatchSource:0}: Error finding container 25cd3bbafac12c9de5648e8837040960f9abf21b39874a8f3c4c37c5d1ec6b56: Status 404 returned error can't find the container with id 25cd3bbafac12c9de5648e8837040960f9abf21b39874a8f3c4c37c5d1ec6b56 Nov 04 01:30:02.612313 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.612278 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz"] Nov 04 01:30:02.615938 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.615871 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:30:02.616043 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.616004 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:30:02.616100 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.616053 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:30:02.746868 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.746802 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz"] Nov 04 01:30:02.748989 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:30:02.748953 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50745197_72c7_41bf_a41e_23a796a4d4d0.slice/crio-f66f11e567ac0347d1f9d0bbeb402cb9dd25ca57c3eda6492b7096f5534984ca WatchSource:0}: Error finding container f66f11e567ac0347d1f9d0bbeb402cb9dd25ca57c3eda6492b7096f5534984ca: Status 404 returned error can't find the container with id f66f11e567ac0347d1f9d0bbeb402cb9dd25ca57c3eda6492b7096f5534984ca Nov 04 01:30:02.759432 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.758740 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:30:02.759432 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.758833 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:30:02.759432 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.758869 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:30:02.975522 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.975482 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" event={"ID":"03ee575f-2616-424f-a7ea-79b03b554a5e","Type":"ContainerStarted","Data":"7407612ad2ee4bfe3974c8d68b3c948c3e444c1d1163343c65d025c826fff2eb"} Nov 04 01:30:02.976193 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.975551 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" event={"ID":"03ee575f-2616-424f-a7ea-79b03b554a5e","Type":"ContainerStarted","Data":"25cd3bbafac12c9de5648e8837040960f9abf21b39874a8f3c4c37c5d1ec6b56"} Nov 04 01:30:02.978881 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.978820 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" event={"ID":"50745197-72c7-41bf-a41e-23a796a4d4d0","Type":"ContainerStarted","Data":"5047e2e49d57d4ea4c8e91764ab840b25f046ceb953c77802a1292f0085f1081"} Nov 04 01:30:02.979124 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:02.978913 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" event={"ID":"50745197-72c7-41bf-a41e-23a796a4d4d0","Type":"ContainerStarted","Data":"f66f11e567ac0347d1f9d0bbeb402cb9dd25ca57c3eda6492b7096f5534984ca"} Nov 04 01:30:03.017052 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.006506 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" podStartSLOduration=2.006487437 podStartE2EDuration="2.006487437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:03.005278414 +0000 UTC m=+1887.947127318" watchObservedRunningTime="2026-11-04 01:30:03.006487437 +0000 UTC m=+1887.948336340" Nov 04 01:30:03.038912 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.038798 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" podStartSLOduration=2.038757864 podStartE2EDuration="2.038757864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:03.027047148 +0000 UTC m=+1887.968896052" watchObservedRunningTime="2026-11-04 01:30:03.038757864 +0000 UTC m=+1887.980606768" Nov 04 01:30:03.281035 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.280944 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:30:03.282850 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.282814 7059 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:30:03.283025 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.282884 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" podUID="03ee575f-2616-424f-a7ea-79b03b554a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 04 01:30:03.326220 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.326178 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:30:03.328494 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.328453 7059 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:30:03.328678 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.328511 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" podUID="50745197-72c7-41bf-a41e-23a796a4d4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 04 01:30:03.777499 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.777465 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:30:03.831461 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.831429 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm"] Nov 04 01:30:03.844048 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.843972 7059 cpu_manager.go:414] "containerMap: removing container" podUID="86db204f-72c0-48a4-817d-34067fbf8f0a" containerName="istio-proxy" Nov 04 01:30:03.860061 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.860016 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86db204f-72c0-48a4-817d-34067fbf8f0a" containerName="istio-proxy" Nov 04 01:30:03.860594 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.860561 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="86db204f-72c0-48a4-817d-34067fbf8f0a" containerName="istio-proxy" Nov 04 01:30:03.865254 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.865086 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-credential-socket\") pod \"86db204f-72c0-48a4-817d-34067fbf8f0a\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " Nov 04 01:30:03.865623 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.865263 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86db204f-72c0-48a4-817d-34067fbf8f0a-istio-token\" (UniqueName: \"kubernetes.io/projected/86db204f-72c0-48a4-817d-34067fbf8f0a-istio-token\") pod \"86db204f-72c0-48a4-817d-34067fbf8f0a\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " Nov 04 01:30:03.865623 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.865307 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86db204f-72c0-48a4-817d-34067fbf8f0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86db204f-72c0-48a4-817d-34067fbf8f0a-istiod-ca-cert\") pod \"86db204f-72c0-48a4-817d-34067fbf8f0a\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " Nov 04 01:30:03.865623 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.865357 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86db204f-72c0-48a4-817d-34067fbf8f0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86db204f-72c0-48a4-817d-34067fbf8f0a-istio-podinfo\") pod \"86db204f-72c0-48a4-817d-34067fbf8f0a\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " Nov 04 01:30:03.865623 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.865436 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86db204f-72c0-48a4-817d-34067fbf8f0a-kube-api-access-7hghn\" (UniqueName: \"kubernetes.io/projected/86db204f-72c0-48a4-817d-34067fbf8f0a-kube-api-access-7hghn\") pod \"86db204f-72c0-48a4-817d-34067fbf8f0a\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " Nov 04 01:30:03.865623 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.865520 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-istio-envoy\") pod \"86db204f-72c0-48a4-817d-34067fbf8f0a\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " Nov 04 01:30:03.876273 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.865625 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-istio-data\") pod \"86db204f-72c0-48a4-817d-34067fbf8f0a\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " Nov 04 01:30:03.876273 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.865715 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-workload-certs\") pod \"86db204f-72c0-48a4-817d-34067fbf8f0a\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " Nov 04 01:30:03.876273 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.865816 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-workload-socket\") pod \"86db204f-72c0-48a4-817d-34067fbf8f0a\" (UID: \"86db204f-72c0-48a4-817d-34067fbf8f0a\") " Nov 04 01:30:03.876273 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.867414 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-istio-data" pod "86db204f-72c0-48a4-817d-34067fbf8f0a" (UID: "86db204f-72c0-48a4-817d-34067fbf8f0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:03.876273 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.867597 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-workload-certs" pod "86db204f-72c0-48a4-817d-34067fbf8f0a" (UID: "86db204f-72c0-48a4-817d-34067fbf8f0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:03.876273 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.867763 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-workload-socket" pod "86db204f-72c0-48a4-817d-34067fbf8f0a" (UID: "86db204f-72c0-48a4-817d-34067fbf8f0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:03.876273 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.868442 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:30:03.876273 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.871009 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-credential-socket" pod "86db204f-72c0-48a4-817d-34067fbf8f0a" (UID: "86db204f-72c0-48a4-817d-34067fbf8f0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:03.876273 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.874128 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86db204f-72c0-48a4-817d-34067fbf8f0a-kube-api-access-7hghn" pod "86db204f-72c0-48a4-817d-34067fbf8f0a" (UID: "86db204f-72c0-48a4-817d-34067fbf8f0a"). InnerVolumeSpecName "kube-api-access-7hghn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:03.876273 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.875110 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86db204f-72c0-48a4-817d-34067fbf8f0a-istiod-ca-cert" pod "86db204f-72c0-48a4-817d-34067fbf8f0a" (UID: "86db204f-72c0-48a4-817d-34067fbf8f0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:03.890462 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.890422 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86db204f-72c0-48a4-817d-34067fbf8f0a-istio-podinfo" pod "86db204f-72c0-48a4-817d-34067fbf8f0a" (UID: "86db204f-72c0-48a4-817d-34067fbf8f0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:30:03.890866 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.890845 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86db204f-72c0-48a4-817d-34067fbf8f0a-istio-token" pod "86db204f-72c0-48a4-817d-34067fbf8f0a" (UID: "86db204f-72c0-48a4-817d-34067fbf8f0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:03.893822 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.893783 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-istio-envoy" pod "86db204f-72c0-48a4-817d-34067fbf8f0a" (UID: "86db204f-72c0-48a4-817d-34067fbf8f0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:03.899058 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.899002 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm"] Nov 04 01:30:03.968621 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.968586 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9m2x\" (UniqueName: \"kubernetes.io/projected/83836442-ba13-44a5-b90d-d3ba7660d2d2-kube-api-access-j9m2x\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:30:03.968756 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.968637 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-istio-envoy\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:30:03.968756 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.968694 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-workload-certs\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:30:03.968843 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.968805 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83836442-ba13-44a5-b90d-d3ba7660d2d2-istiod-ca-cert\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:30:03.968843 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.968827 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83836442-ba13-44a5-b90d-d3ba7660d2d2-istio-podinfo\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:30:03.968999 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.968970 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-workload-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:30:03.969131 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.969071 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83836442-ba13-44a5-b90d-d3ba7660d2d2-istio-token\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:30:03.969183 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.969139 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-credential-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:30:03.969272 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.969258 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-istio-data\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:30:03.969431 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.969420 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:03.969487 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.969434 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86db204f-72c0-48a4-817d-34067fbf8f0a-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:03.969487 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.969444 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86db204f-72c0-48a4-817d-34067fbf8f0a-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:03.969487 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.969453 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86db204f-72c0-48a4-817d-34067fbf8f0a-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:03.969487 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.969464 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hghn\" (UniqueName: \"kubernetes.io/projected/86db204f-72c0-48a4-817d-34067fbf8f0a-kube-api-access-7hghn\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:03.969487 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.969477 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:03.969487 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.969486 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:03.969759 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.969496 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:03.969759 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.969505 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86db204f-72c0-48a4-817d-34067fbf8f0a-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:03.990310 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.990277 7059 generic.go:356] "Generic (PLEG): container finished" podID="86db204f-72c0-48a4-817d-34067fbf8f0a" containerID="ac60ec4fb6d14ad97c1911e6248151c036c2f4584abdb302a82d8ce69e8c9b4f" exitCode=0 Nov 04 01:30:03.990791 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.990332 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" Nov 04 01:30:03.990791 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.990391 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" event={"ID":"86db204f-72c0-48a4-817d-34067fbf8f0a","Type":"ContainerDied","Data":"ac60ec4fb6d14ad97c1911e6248151c036c2f4584abdb302a82d8ce69e8c9b4f"} Nov 04 01:30:03.990791 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.990489 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll" event={"ID":"86db204f-72c0-48a4-817d-34067fbf8f0a","Type":"ContainerDied","Data":"05b3b63ea597158d88011dadbfb5d9f9d3b577cff3519509145a482bb2bab735"} Nov 04 01:30:03.990791 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:03.990551 7059 scope.go:122] "RemoveContainer" containerID="ac60ec4fb6d14ad97c1911e6248151c036c2f4584abdb302a82d8ce69e8c9b4f" Nov 04 01:30:04.004207 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:04.004188 7059 scope.go:122] "RemoveContainer" containerID="ac60ec4fb6d14ad97c1911e6248151c036c2f4584abdb302a82d8ce69e8c9b4f" Nov 04 01:30:04.004512 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:30:04.004494 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac60ec4fb6d14ad97c1911e6248151c036c2f4584abdb302a82d8ce69e8c9b4f\": container with ID starting with ac60ec4fb6d14ad97c1911e6248151c036c2f4584abdb302a82d8ce69e8c9b4f not found: ID does not exist" containerID="ac60ec4fb6d14ad97c1911e6248151c036c2f4584abdb302a82d8ce69e8c9b4f" Nov 04 01:30:04.004560 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:04.004522 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac60ec4fb6d14ad97c1911e6248151c036c2f4584abdb302a82d8ce69e8c9b4f"} err="failed to get container status \"ac60ec4fb6d14ad97c1911e6248151c036c2f4584abdb302a82d8ce69e8c9b4f\": rpc error: code = NotFound desc = could not find container \"ac60ec4fb6d14ad97c1911e6248151c036c2f4584abdb302a82d8ce69e8c9b4f\": container with ID starting with ac60ec4fb6d14ad97c1911e6248151c036c2f4584abdb302a82d8ce69e8c9b4f not found: ID does not exist" Nov 04 01:30:04.029165 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:04.029141 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll"] Nov 04 01:30:04.034884 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:04.034862 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8qrll"] Nov 04 01:30:04.071360 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:04.071329 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-workload-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:30:04.071501 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:04.071444 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83836442-ba13-44a5-b90d-d3ba7660d2d2-istio-token\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:30:04.071660 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:04.071636 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-credential-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:30:04.071765 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:04.071743 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-workload-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:30:04.072068 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:04.072046 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-credential-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:30:04.072068 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:04.072050 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-istio-data\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:30:04.072259 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:04.072240 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9m2x\" (UniqueName: \"kubernetes.io/projected/83836442-ba13-44a5-b90d-d3ba7660d2d2-kube-api-access-j9m2x\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:30:04.072305 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:04.072289 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-istio-data\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:30:04.072341 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:04.072295 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-istio-envoy\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:30:04.072377 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:04.072345 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-workload-certs\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:30:04.072484 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:04.072468 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83836442-ba13-44a5-b90d-d3ba7660d2d2-istiod-ca-cert\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:30:04.072521 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:04.072505 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83836442-ba13-44a5-b90d-d3ba7660d2d2-istio-podinfo\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:30:04.072740 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:04.072719 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-workload-certs\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:30:04.073095 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:04.073076 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83836442-ba13-44a5-b90d-d3ba7660d2d2-istiod-ca-cert\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:30:04.074636 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:04.074609 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-istio-envoy\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:30:04.075114 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:04.075094 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83836442-ba13-44a5-b90d-d3ba7660d2d2-istio-podinfo\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:30:04.080703 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:04.080680 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83836442-ba13-44a5-b90d-d3ba7660d2d2-istio-token\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:30:04.081844 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:04.081817 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9m2x\" (UniqueName: \"kubernetes.io/projected/83836442-ba13-44a5-b90d-d3ba7660d2d2-kube-api-access-j9m2x\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:30:04.216743 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:04.216712 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:30:04.283200 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:04.282593 7059 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:30:04.283200 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:04.282649 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" podUID="03ee575f-2616-424f-a7ea-79b03b554a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 04 01:30:04.328268 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:04.327333 7059 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:30:04.328268 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:04.327404 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" podUID="50745197-72c7-41bf-a41e-23a796a4d4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 04 01:30:04.530639 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:30:04.530608 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83836442_ba13_44a5_b90d_d3ba7660d2d2.slice/crio-9159ad10ae66630478351d66b45c7f49c489e2863a2e370df7a9ba7d530fb83c WatchSource:0}: Error finding container 9159ad10ae66630478351d66b45c7f49c489e2863a2e370df7a9ba7d530fb83c: Status 404 returned error can't find the container with id 9159ad10ae66630478351d66b45c7f49c489e2863a2e370df7a9ba7d530fb83c Nov 04 01:30:04.533459 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:04.533433 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm"] Nov 04 01:30:04.534464 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:04.534427 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:30:04.534558 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:04.534507 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:30:04.534558 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:04.534550 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:30:05.083424 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:05.083373 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" event={"ID":"83836442-ba13-44a5-b90d-d3ba7660d2d2","Type":"ContainerStarted","Data":"061a50b2340a540fae58b082467ff0c35adaa264eb3fc001853d4d9feb41f26f"} Nov 04 01:30:05.084202 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:05.084172 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" event={"ID":"83836442-ba13-44a5-b90d-d3ba7660d2d2","Type":"ContainerStarted","Data":"9159ad10ae66630478351d66b45c7f49c489e2863a2e370df7a9ba7d530fb83c"} Nov 04 01:30:05.131084 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:05.131023 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" podStartSLOduration=4.131005198 podStartE2EDuration="4.131005198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:05.127766237 +0000 UTC m=+1890.069615141" watchObservedRunningTime="2026-11-04 01:30:05.131005198 +0000 UTC m=+1890.072854096" Nov 04 01:30:05.223518 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:05.223485 7059 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:30:05.223686 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:05.223532 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" podUID="83836442-ba13-44a5-b90d-d3ba7660d2d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 04 01:30:05.231321 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:05.231295 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86db204f-72c0-48a4-817d-34067fbf8f0a" path="/var/lib/kubelet/pods/86db204f-72c0-48a4-817d-34067fbf8f0a/volumes" Nov 04 01:30:05.231657 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:05.231644 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:30:05.285954 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:05.281424 7059 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:30:05.285954 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:05.281485 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" podUID="03ee575f-2616-424f-a7ea-79b03b554a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 04 01:30:05.327501 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:05.326116 7059 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:30:05.327501 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:05.326171 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" podUID="50745197-72c7-41bf-a41e-23a796a4d4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 04 01:30:06.220949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:06.220370 7059 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:30:06.220949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:06.220434 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" podUID="83836442-ba13-44a5-b90d-d3ba7660d2d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 04 01:30:06.283362 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:06.283320 7059 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:30:06.283615 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:06.283585 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" podUID="03ee575f-2616-424f-a7ea-79b03b554a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 04 01:30:06.326469 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:06.326422 7059 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:30:06.326674 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:06.326478 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" podUID="50745197-72c7-41bf-a41e-23a796a4d4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 04 01:30:06.919840 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:30:06.900227 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:30:07.304951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:07.304292 7059 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:30:07.304951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:07.304356 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" podUID="83836442-ba13-44a5-b90d-d3ba7660d2d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 04 01:30:07.304951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:07.304292 7059 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:30:07.304951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:07.304891 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" podUID="03ee575f-2616-424f-a7ea-79b03b554a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 04 01:30:07.330134 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:07.330015 7059 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:30:07.330351 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:07.330140 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" podUID="50745197-72c7-41bf-a41e-23a796a4d4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 04 01:30:07.673013 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:07.661025 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee876a-54f4fd798c-djxkh" podUID="a70660e3-f657-4f28-9743-651390c8ae3f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:30:07.837894 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:07.831244 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 04 01:30:07.837894 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:07.836859 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded" Nov 04 01:30:07.947056 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:07.946965 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:30:08.218024 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:08.217354 7059 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:30:08.218024 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:08.217422 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" podUID="83836442-ba13-44a5-b90d-d3ba7660d2d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 04 01:30:08.218024 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:08.217847 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm" podUID="abeae128-5b8c-4d5e-9592-f64e9c6bd4b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:30:08.287054 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:08.287016 7059 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:30:08.287325 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:08.287295 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" podUID="03ee575f-2616-424f-a7ea-79b03b554a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 04 01:30:08.329653 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:08.329618 7059 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:30:08.330285 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:08.330250 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" podUID="50745197-72c7-41bf-a41e-23a796a4d4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 04 01:30:08.389097 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:08.389030 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:30:09.230254 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:09.229632 7059 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:30:09.230254 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:09.229698 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" podUID="83836442-ba13-44a5-b90d-d3ba7660d2d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 04 01:30:09.314951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:09.292168 7059 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:30:09.314951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:09.292226 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" podUID="03ee575f-2616-424f-a7ea-79b03b554a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 04 01:30:09.390032 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:09.389363 7059 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:30:09.390032 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:09.389429 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" podUID="50745197-72c7-41bf-a41e-23a796a4d4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 04 01:30:09.435515 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:30:09.419501 7059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70660e3_f657_4f28_9743_651390c8ae3f.slice/crio-1859918638a2b822c70df32a32e0fcc5e1ad34c9d4f31488d28724f7c051a4ed.scope\": RecentStats: unable to find data in memory cache]" Nov 04 01:30:09.636092 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:09.635837 7059 generic.go:356] "Generic (PLEG): container finished" podID="a70660e3-f657-4f28-9743-651390c8ae3f" containerID="1859918638a2b822c70df32a32e0fcc5e1ad34c9d4f31488d28724f7c051a4ed" exitCode=0 Nov 04 01:30:09.636092 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:09.635970 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee876a-54f4fd798c-djxkh" event={"ID":"a70660e3-f657-4f28-9743-651390c8ae3f","Type":"ContainerDied","Data":"1859918638a2b822c70df32a32e0fcc5e1ad34c9d4f31488d28724f7c051a4ed"} Nov 04 01:30:10.389873 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:10.374046 7059 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:30:10.389873 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:10.374110 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" podUID="03ee575f-2616-424f-a7ea-79b03b554a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 04 01:30:10.389873 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:10.374524 7059 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:30:10.389873 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:10.374561 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" podUID="50745197-72c7-41bf-a41e-23a796a4d4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 04 01:30:10.402364 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:10.401067 7059 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:30:10.402364 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:10.401143 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" podUID="83836442-ba13-44a5-b90d-d3ba7660d2d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 04 01:30:10.411621 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:10.411163 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee876a-54f4fd798c-djxkh" Nov 04 01:30:10.536747 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:10.536706 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a70660e3-f657-4f28-9743-651390c8ae3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a70660e3-f657-4f28-9743-651390c8ae3f-plugins-config-volume\") pod \"a70660e3-f657-4f28-9743-651390c8ae3f\" (UID: \"a70660e3-f657-4f28-9743-651390c8ae3f\") " Nov 04 01:30:10.561874 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:10.561480 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a70660e3-f657-4f28-9743-651390c8ae3f-kube-api-access-4bwrx\" (UniqueName: \"kubernetes.io/projected/a70660e3-f657-4f28-9743-651390c8ae3f-kube-api-access-4bwrx\") pod \"a70660e3-f657-4f28-9743-651390c8ae3f\" (UID: \"a70660e3-f657-4f28-9743-651390c8ae3f\") " Nov 04 01:30:10.571763 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:10.548494 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a70660e3-f657-4f28-9743-651390c8ae3f-plugins-config-volume" pod "a70660e3-f657-4f28-9743-651390c8ae3f" (UID: "a70660e3-f657-4f28-9743-651390c8ae3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:10.584957 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:10.584903 7059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a70660e3-f657-4f28-9743-651390c8ae3f-plugins-config-volume\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:10.607816 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:10.607784 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a70660e3-f657-4f28-9743-651390c8ae3f-kube-api-access-4bwrx" pod "a70660e3-f657-4f28-9743-651390c8ae3f" (UID: "a70660e3-f657-4f28-9743-651390c8ae3f"). InnerVolumeSpecName "kube-api-access-4bwrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:10.716689 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:10.716650 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bwrx\" (UniqueName: \"kubernetes.io/projected/a70660e3-f657-4f28-9743-651390c8ae3f-kube-api-access-4bwrx\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:10.727625 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:10.727593 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee876a-54f4fd798c-djxkh" Nov 04 01:30:10.727625 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:10.727608 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee876a-54f4fd798c-djxkh" event={"ID":"a70660e3-f657-4f28-9743-651390c8ae3f","Type":"ContainerDied","Data":"b2bdd8787b27ee3afd9f49043bba23c71d86c095705a5cdb7eb0029aad5d0efc"} Nov 04 01:30:10.727903 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:10.727672 7059 scope.go:122] "RemoveContainer" containerID="1859918638a2b822c70df32a32e0fcc5e1ad34c9d4f31488d28724f7c051a4ed" Nov 04 01:30:10.832940 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:10.829036 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ee876a-54f4fd798c-djxkh"] Nov 04 01:30:10.832940 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:10.831279 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ee876a-54f4fd798c-djxkh"] Nov 04 01:30:11.237178 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:11.237142 7059 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:30:11.280972 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:11.280908 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" podUID="83836442-ba13-44a5-b90d-d3ba7660d2d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 04 01:30:11.322861 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:11.285658 7059 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:30:11.322861 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:11.322805 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" podUID="03ee575f-2616-424f-a7ea-79b03b554a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 04 01:30:11.401971 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:11.343047 7059 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:30:11.401971 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:11.343120 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" podUID="50745197-72c7-41bf-a41e-23a796a4d4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 04 01:30:11.605005 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:11.603980 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a70660e3-f657-4f28-9743-651390c8ae3f" path="/var/lib/kubelet/pods/a70660e3-f657-4f28-9743-651390c8ae3f/volumes" Nov 04 01:30:11.605005 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:11.604480 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-30b1a1-776c9d9d59-tlcl7"] Nov 04 01:30:11.609016 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:11.605856 7059 cpu_manager.go:414] "containerMap: removing container" podUID="a70660e3-f657-4f28-9743-651390c8ae3f" containerName="payload-pre-processing" Nov 04 01:30:11.609016 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:11.605881 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a70660e3-f657-4f28-9743-651390c8ae3f" containerName="payload-pre-processing" Nov 04 01:30:11.609016 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:11.606349 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="a70660e3-f657-4f28-9743-651390c8ae3f" containerName="payload-pre-processing" Nov 04 01:30:11.612604 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:11.612571 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-30b1a1-84477cd7fd-kdnpv"] Nov 04 01:30:11.614009 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:11.613983 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30b1a1-776c9d9d59-tlcl7" Nov 04 01:30:11.619653 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:11.619617 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-30b1a1\"" Nov 04 01:30:11.620039 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:11.620007 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-30b1a1-dockercfg-lwwqz\"" Nov 04 01:30:11.620271 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:11.620250 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-30b1a1-776c9d9d59-tlcl7"] Nov 04 01:30:11.620358 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:11.620322 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-30b1a1-84477cd7fd-kdnpv"] Nov 04 01:30:11.620420 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:11.620410 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-30b1a1-84477cd7fd-kdnpv" Nov 04 01:30:12.161947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:12.142298 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a786b7b2-9cad-4538-844a-99fa4a135fd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-30b1a1-776c9d9d59-tlcl7\" (UID: \"a786b7b2-9cad-4538-844a-99fa4a135fd3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-30b1a1-776c9d9d59-tlcl7" Nov 04 01:30:12.161947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:12.142535 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d7439cc-59c5-496c-8577-fb895bba5a63-plugins-config-volume\") pod \"payload-processing-e2e-api-a-30b1a1-84477cd7fd-kdnpv\" (UID: \"7d7439cc-59c5-496c-8577-fb895bba5a63\") " pod="openshift-ingress/payload-processing-e2e-api-a-30b1a1-84477cd7fd-kdnpv" Nov 04 01:30:12.161947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:12.142723 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb4qh\" (UniqueName: \"kubernetes.io/projected/a786b7b2-9cad-4538-844a-99fa4a135fd3-kube-api-access-vb4qh\") pod \"payload-pre-processing-e2e-api-a-30b1a1-776c9d9d59-tlcl7\" (UID: \"a786b7b2-9cad-4538-844a-99fa4a135fd3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-30b1a1-776c9d9d59-tlcl7" Nov 04 01:30:12.161947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:12.142811 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqlt8\" (UniqueName: \"kubernetes.io/projected/7d7439cc-59c5-496c-8577-fb895bba5a63-kube-api-access-rqlt8\") pod \"payload-processing-e2e-api-a-30b1a1-84477cd7fd-kdnpv\" (UID: \"7d7439cc-59c5-496c-8577-fb895bba5a63\") " pod="openshift-ingress/payload-processing-e2e-api-a-30b1a1-84477cd7fd-kdnpv" Nov 04 01:30:12.290281 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:12.290247 7059 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:30:12.290566 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:12.290524 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" podUID="83836442-ba13-44a5-b90d-d3ba7660d2d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 04 01:30:12.350300 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:12.330628 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb4qh\" (UniqueName: \"kubernetes.io/projected/a786b7b2-9cad-4538-844a-99fa4a135fd3-kube-api-access-vb4qh\") pod \"payload-pre-processing-e2e-api-a-30b1a1-776c9d9d59-tlcl7\" (UID: \"a786b7b2-9cad-4538-844a-99fa4a135fd3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-30b1a1-776c9d9d59-tlcl7" Nov 04 01:30:12.350300 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:12.330724 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqlt8\" (UniqueName: \"kubernetes.io/projected/7d7439cc-59c5-496c-8577-fb895bba5a63-kube-api-access-rqlt8\") pod \"payload-processing-e2e-api-a-30b1a1-84477cd7fd-kdnpv\" (UID: \"7d7439cc-59c5-496c-8577-fb895bba5a63\") " pod="openshift-ingress/payload-processing-e2e-api-a-30b1a1-84477cd7fd-kdnpv" Nov 04 01:30:12.350300 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:12.330869 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a786b7b2-9cad-4538-844a-99fa4a135fd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-30b1a1-776c9d9d59-tlcl7\" (UID: \"a786b7b2-9cad-4538-844a-99fa4a135fd3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-30b1a1-776c9d9d59-tlcl7" Nov 04 01:30:12.350300 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:12.331018 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d7439cc-59c5-496c-8577-fb895bba5a63-plugins-config-volume\") pod \"payload-processing-e2e-api-a-30b1a1-84477cd7fd-kdnpv\" (UID: \"7d7439cc-59c5-496c-8577-fb895bba5a63\") " pod="openshift-ingress/payload-processing-e2e-api-a-30b1a1-84477cd7fd-kdnpv" Nov 04 01:30:12.350300 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:12.332579 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d7439cc-59c5-496c-8577-fb895bba5a63-plugins-config-volume\") pod \"payload-processing-e2e-api-a-30b1a1-84477cd7fd-kdnpv\" (UID: \"7d7439cc-59c5-496c-8577-fb895bba5a63\") " pod="openshift-ingress/payload-processing-e2e-api-a-30b1a1-84477cd7fd-kdnpv" Nov 04 01:30:12.350300 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:12.334395 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a786b7b2-9cad-4538-844a-99fa4a135fd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-30b1a1-776c9d9d59-tlcl7\" (UID: \"a786b7b2-9cad-4538-844a-99fa4a135fd3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-30b1a1-776c9d9d59-tlcl7" Nov 04 01:30:12.409469 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:12.409431 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:30:12.409958 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:12.409563 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:30:12.413286 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:12.413224 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:30:12.413286 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:12.413268 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:30:12.416228 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:12.416171 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:30:12.416350 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:12.416338 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:30:12.431523 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:12.431480 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb4qh\" (UniqueName: \"kubernetes.io/projected/a786b7b2-9cad-4538-844a-99fa4a135fd3-kube-api-access-vb4qh\") pod \"payload-pre-processing-e2e-api-a-30b1a1-776c9d9d59-tlcl7\" (UID: \"a786b7b2-9cad-4538-844a-99fa4a135fd3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-30b1a1-776c9d9d59-tlcl7" Nov 04 01:30:12.440382 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:12.439308 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqlt8\" (UniqueName: \"kubernetes.io/projected/7d7439cc-59c5-496c-8577-fb895bba5a63-kube-api-access-rqlt8\") pod \"payload-processing-e2e-api-a-30b1a1-84477cd7fd-kdnpv\" (UID: \"7d7439cc-59c5-496c-8577-fb895bba5a63\") " pod="openshift-ingress/payload-processing-e2e-api-a-30b1a1-84477cd7fd-kdnpv" Nov 04 01:30:12.539361 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:12.536094 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30b1a1-776c9d9d59-tlcl7" Nov 04 01:30:12.559026 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:12.558226 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-30b1a1-84477cd7fd-kdnpv" Nov 04 01:30:12.928537 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:12.928502 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-30b1a1-776c9d9d59-tlcl7"] Nov 04 01:30:12.929190 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:30:12.929143 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda786b7b2_9cad_4538_844a_99fa4a135fd3.slice/crio-6e354b6a2d780414d28d329f6c415855110e9833a071131fafc50291bcd56d74 WatchSource:0}: Error finding container 6e354b6a2d780414d28d329f6c415855110e9833a071131fafc50291bcd56d74: Status 404 returned error can't find the container with id 6e354b6a2d780414d28d329f6c415855110e9833a071131fafc50291bcd56d74 Nov 04 01:30:12.960575 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:12.960545 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-30b1a1-84477cd7fd-kdnpv"] Nov 04 01:30:12.978343 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:30:12.978316 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d7439cc_59c5_496c_8577_fb895bba5a63.slice/crio-505d9ab96f87175221805971fdf6e5111307e9d72e73a9be72233f7aca04776d WatchSource:0}: Error finding container 505d9ab96f87175221805971fdf6e5111307e9d72e73a9be72233f7aca04776d: Status 404 returned error can't find the container with id 505d9ab96f87175221805971fdf6e5111307e9d72e73a9be72233f7aca04776d Nov 04 01:30:13.221577 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:13.221497 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:30:13.221789 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:13.221767 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:30:13.222385 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:13.222364 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:30:13.405466 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:13.405431 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30b1a1-776c9d9d59-tlcl7" event={"ID":"a786b7b2-9cad-4538-844a-99fa4a135fd3","Type":"ContainerStarted","Data":"b37a2f851f0d365edffd79913a94f51fd114904420b8ceab7b12eb3465b22409"} Nov 04 01:30:13.405668 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:13.405497 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30b1a1-776c9d9d59-tlcl7" event={"ID":"a786b7b2-9cad-4538-844a-99fa4a135fd3","Type":"ContainerStarted","Data":"6e354b6a2d780414d28d329f6c415855110e9833a071131fafc50291bcd56d74"} Nov 04 01:30:13.405668 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:13.405530 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30b1a1-776c9d9d59-tlcl7" Nov 04 01:30:13.407410 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:13.407373 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-30b1a1-84477cd7fd-kdnpv" event={"ID":"7d7439cc-59c5-496c-8577-fb895bba5a63","Type":"ContainerStarted","Data":"58350508a605077f2c3cebe1edece0591fc4832b94a3ecf38878bd3bd53c1dd7"} Nov 04 01:30:13.407535 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:13.407433 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-30b1a1-84477cd7fd-kdnpv" Nov 04 01:30:13.407535 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:13.407455 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-30b1a1-84477cd7fd-kdnpv" event={"ID":"7d7439cc-59c5-496c-8577-fb895bba5a63","Type":"ContainerStarted","Data":"505d9ab96f87175221805971fdf6e5111307e9d72e73a9be72233f7aca04776d"} Nov 04 01:30:13.430477 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:13.430423 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30b1a1-776c9d9d59-tlcl7" podStartSLOduration=2.430405727 podStartE2EDuration="2.430405727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:13.426239418 +0000 UTC m=+1898.368088319" watchObservedRunningTime="2026-11-04 01:30:13.430405727 +0000 UTC m=+1898.372254633" Nov 04 01:30:13.457805 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:13.457748 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-30b1a1-84477cd7fd-kdnpv" podStartSLOduration=2.457731663 podStartE2EDuration="2.457731663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:13.450988138 +0000 UTC m=+1898.392837040" watchObservedRunningTime="2026-11-04 01:30:13.457731663 +0000 UTC m=+1898.399580567" Nov 04 01:30:21.916523 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:21.916433 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62"] Nov 04 01:30:21.917047 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:21.916677 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" podUID="60fe9f56-ebb0-4b4c-b325-8f3cb73c029c" containerName="istio-proxy" containerID="cri-o://54b2d67c46b6afe96124bc1cca8e4ad7a996323280e531b3967a482cba02bdd1" gracePeriod=30 Nov 04 01:30:21.927639 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:21.927612 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c"] Nov 04 01:30:21.927848 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:21.927818 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" podUID="19a17fd1-42d6-473d-9568-ad6ba68ca22a" containerName="istio-proxy" containerID="cri-o://8839dc00f2ba41a9fece0559748d6b6bababd2f2a5f4032adc24a555f48c5c51" gracePeriod=30 Nov 04 01:30:21.937166 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:21.937121 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp"] Nov 04 01:30:21.937390 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:21.937347 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" podUID="8d6194a6-04a4-4d8d-8017-de919dc36fa5" containerName="istio-proxy" containerID="cri-o://89d5d9573cb883da8e2894ebc53363f63ee8d469a14dbfbdc8e7a1c3ac69ab71" gracePeriod=30 Nov 04 01:30:24.177224 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:24.177185 7059 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:30:24.177628 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:24.177276 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" podUID="19a17fd1-42d6-473d-9568-ad6ba68ca22a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:30:24.419815 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:24.419781 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30b1a1-776c9d9d59-tlcl7" Nov 04 01:30:24.420026 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:24.419867 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-30b1a1-84477cd7fd-kdnpv" Nov 04 01:30:27.298208 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.298170 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:30:27.372102 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.366634 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istiod-ca-cert\") pod \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " Nov 04 01:30:27.372102 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.366805 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-credential-socket\") pod \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " Nov 04 01:30:27.372102 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.366859 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-workload-socket\") pod \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " Nov 04 01:30:27.372102 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.366946 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istio-data\") pod \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " Nov 04 01:30:27.372102 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.367125 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-workload-certs\") pod \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " Nov 04 01:30:27.372102 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.367212 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19a17fd1-42d6-473d-9568-ad6ba68ca22a-kube-api-access-kj9ft\" (UniqueName: \"kubernetes.io/projected/19a17fd1-42d6-473d-9568-ad6ba68ca22a-kube-api-access-kj9ft\") pod \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " Nov 04 01:30:27.380041 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.377464 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istio-token\" (UniqueName: \"kubernetes.io/projected/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istio-token\") pod \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " Nov 04 01:30:27.380041 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.377815 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istio-envoy\") pod \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " Nov 04 01:30:27.380041 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.377862 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istio-podinfo\") pod \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\" (UID: \"19a17fd1-42d6-473d-9568-ad6ba68ca22a\") " Nov 04 01:30:27.393435 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.393389 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istiod-ca-cert" pod "19a17fd1-42d6-473d-9568-ad6ba68ca22a" (UID: "19a17fd1-42d6-473d-9568-ad6ba68ca22a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:27.393711 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.393688 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-credential-socket" pod "19a17fd1-42d6-473d-9568-ad6ba68ca22a" (UID: "19a17fd1-42d6-473d-9568-ad6ba68ca22a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:27.393878 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.393860 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-workload-socket" pod "19a17fd1-42d6-473d-9568-ad6ba68ca22a" (UID: "19a17fd1-42d6-473d-9568-ad6ba68ca22a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:27.394081 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.394055 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istio-data" pod "19a17fd1-42d6-473d-9568-ad6ba68ca22a" (UID: "19a17fd1-42d6-473d-9568-ad6ba68ca22a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:27.394568 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.394247 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-workload-certs" pod "19a17fd1-42d6-473d-9568-ad6ba68ca22a" (UID: "19a17fd1-42d6-473d-9568-ad6ba68ca22a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:27.457370 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.457313 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istio-envoy" pod "19a17fd1-42d6-473d-9568-ad6ba68ca22a" (UID: "19a17fd1-42d6-473d-9568-ad6ba68ca22a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:27.457848 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.457820 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a17fd1-42d6-473d-9568-ad6ba68ca22a-kube-api-access-kj9ft" pod "19a17fd1-42d6-473d-9568-ad6ba68ca22a" (UID: "19a17fd1-42d6-473d-9568-ad6ba68ca22a"). InnerVolumeSpecName "kube-api-access-kj9ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:27.460575 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.460507 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istio-token" pod "19a17fd1-42d6-473d-9568-ad6ba68ca22a" (UID: "19a17fd1-42d6-473d-9568-ad6ba68ca22a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:27.465596 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.464839 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istio-podinfo" pod "19a17fd1-42d6-473d-9568-ad6ba68ca22a" (UID: "19a17fd1-42d6-473d-9568-ad6ba68ca22a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:30:27.481032 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.480874 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.481032 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.480898 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.481032 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.480908 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.481032 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.480916 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.481032 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.480949 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.481032 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.480962 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.481032 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.480974 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19a17fd1-42d6-473d-9568-ad6ba68ca22a-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.481032 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.480987 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj9ft\" (UniqueName: \"kubernetes.io/projected/19a17fd1-42d6-473d-9568-ad6ba68ca22a-kube-api-access-kj9ft\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.481032 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.481002 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19a17fd1-42d6-473d-9568-ad6ba68ca22a-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.486011 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.485984 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:30:27.495599 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.495572 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:30:27.581686 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.581647 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istiod-ca-cert\") pod \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " Nov 04 01:30:27.581877 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.581700 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-kube-api-access-sxnrr\" (UniqueName: \"kubernetes.io/projected/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-kube-api-access-sxnrr\") pod \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " Nov 04 01:30:27.581877 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.581748 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d6194a6-04a4-4d8d-8017-de919dc36fa5-kube-api-access-twbdx\" (UniqueName: \"kubernetes.io/projected/8d6194a6-04a4-4d8d-8017-de919dc36fa5-kube-api-access-twbdx\") pod \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " Nov 04 01:30:27.581877 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.581775 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-workload-socket\") pod \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " Nov 04 01:30:27.582114 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.581905 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istio-token\" (UniqueName: \"kubernetes.io/projected/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istio-token\") pod \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " Nov 04 01:30:27.582114 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.582016 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istio-data\") pod \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " Nov 04 01:30:27.582249 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.582112 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istio-podinfo\") pod \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " Nov 04 01:30:27.582249 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.582143 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istio-token\" (UniqueName: \"kubernetes.io/projected/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istio-token\") pod \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " Nov 04 01:30:27.582249 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.582141 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-workload-socket" pod "8d6194a6-04a4-4d8d-8017-de919dc36fa5" (UID: "8d6194a6-04a4-4d8d-8017-de919dc36fa5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:27.582249 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.582199 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istiod-ca-cert" pod "60fe9f56-ebb0-4b4c-b325-8f3cb73c029c" (UID: "60fe9f56-ebb0-4b4c-b325-8f3cb73c029c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:27.582459 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.582315 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-credential-socket\") pod \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " Nov 04 01:30:27.582459 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.582386 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-credential-socket\") pod \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " Nov 04 01:30:27.582459 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.582432 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istio-podinfo\") pod \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " Nov 04 01:30:27.582459 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.582443 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istio-data" pod "8d6194a6-04a4-4d8d-8017-de919dc36fa5" (UID: "8d6194a6-04a4-4d8d-8017-de919dc36fa5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:27.582668 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.582516 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-workload-socket\") pod \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " Nov 04 01:30:27.582668 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.582606 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-workload-certs\") pod \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " Nov 04 01:30:27.582668 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.582652 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-workload-certs\") pod \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " Nov 04 01:30:27.582668 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.582656 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-credential-socket" pod "60fe9f56-ebb0-4b4c-b325-8f3cb73c029c" (UID: "60fe9f56-ebb0-4b4c-b325-8f3cb73c029c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:27.582886 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.582683 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istiod-ca-cert\") pod \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " Nov 04 01:30:27.582886 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.582755 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istio-envoy\") pod \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\" (UID: \"8d6194a6-04a4-4d8d-8017-de919dc36fa5\") " Nov 04 01:30:27.582886 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.582823 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istio-envoy\") pod \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " Nov 04 01:30:27.582886 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.582873 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-workload-certs" pod "8d6194a6-04a4-4d8d-8017-de919dc36fa5" (UID: "8d6194a6-04a4-4d8d-8017-de919dc36fa5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:27.583171 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.583097 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-workload-socket" pod "60fe9f56-ebb0-4b4c-b325-8f3cb73c029c" (UID: "60fe9f56-ebb0-4b4c-b325-8f3cb73c029c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:27.583171 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.583103 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istio-data\") pod \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\" (UID: \"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c\") " Nov 04 01:30:27.583323 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.583300 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istio-data" pod "60fe9f56-ebb0-4b4c-b325-8f3cb73c029c" (UID: "60fe9f56-ebb0-4b4c-b325-8f3cb73c029c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:27.583759 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.583733 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istiod-ca-cert" pod "8d6194a6-04a4-4d8d-8017-de919dc36fa5" (UID: "8d6194a6-04a4-4d8d-8017-de919dc36fa5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:27.584053 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.584029 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-workload-certs" pod "60fe9f56-ebb0-4b4c-b325-8f3cb73c029c" (UID: "60fe9f56-ebb0-4b4c-b325-8f3cb73c029c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:27.584243 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.584222 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-credential-socket" pod "8d6194a6-04a4-4d8d-8017-de919dc36fa5" (UID: "8d6194a6-04a4-4d8d-8017-de919dc36fa5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:27.584576 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.584554 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.584692 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.584678 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.584786 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.584775 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.584957 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.584942 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.585048 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.585037 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.585133 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.585122 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.585217 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.585203 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.585377 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.585363 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.585494 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.585481 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.585578 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.585562 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.585578 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.585281 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istio-podinfo" pod "60fe9f56-ebb0-4b4c-b325-8f3cb73c029c" (UID: "60fe9f56-ebb0-4b4c-b325-8f3cb73c029c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:30:27.585985 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.585961 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d6194a6-04a4-4d8d-8017-de919dc36fa5-kube-api-access-twbdx" pod "8d6194a6-04a4-4d8d-8017-de919dc36fa5" (UID: "8d6194a6-04a4-4d8d-8017-de919dc36fa5"). InnerVolumeSpecName "kube-api-access-twbdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:27.588559 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.588533 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istio-envoy" pod "8d6194a6-04a4-4d8d-8017-de919dc36fa5" (UID: "8d6194a6-04a4-4d8d-8017-de919dc36fa5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:27.589110 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.589077 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istio-podinfo" pod "8d6194a6-04a4-4d8d-8017-de919dc36fa5" (UID: "8d6194a6-04a4-4d8d-8017-de919dc36fa5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:30:27.589205 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.589134 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istio-token" pod "8d6194a6-04a4-4d8d-8017-de919dc36fa5" (UID: "8d6194a6-04a4-4d8d-8017-de919dc36fa5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:27.589205 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.589172 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-kube-api-access-sxnrr" pod "60fe9f56-ebb0-4b4c-b325-8f3cb73c029c" (UID: "60fe9f56-ebb0-4b4c-b325-8f3cb73c029c"). InnerVolumeSpecName "kube-api-access-sxnrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:27.589678 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.589657 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istio-token" pod "60fe9f56-ebb0-4b4c-b325-8f3cb73c029c" (UID: "60fe9f56-ebb0-4b4c-b325-8f3cb73c029c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:27.590497 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.590477 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istio-envoy" pod "60fe9f56-ebb0-4b4c-b325-8f3cb73c029c" (UID: "60fe9f56-ebb0-4b4c-b325-8f3cb73c029c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:27.607580 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.607544 7059 generic.go:356] "Generic (PLEG): container finished" podID="19a17fd1-42d6-473d-9568-ad6ba68ca22a" containerID="8839dc00f2ba41a9fece0559748d6b6bababd2f2a5f4032adc24a555f48c5c51" exitCode=0 Nov 04 01:30:27.608033 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.608005 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" event={"ID":"19a17fd1-42d6-473d-9568-ad6ba68ca22a","Type":"ContainerDied","Data":"8839dc00f2ba41a9fece0559748d6b6bababd2f2a5f4032adc24a555f48c5c51"} Nov 04 01:30:27.608244 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.608224 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" event={"ID":"19a17fd1-42d6-473d-9568-ad6ba68ca22a","Type":"ContainerDied","Data":"297dcccedcf1e2c5de11f28d6c8cffb0b5455fbde343b577706866e931ada778"} Nov 04 01:30:27.608453 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.608432 7059 scope.go:122] "RemoveContainer" containerID="8839dc00f2ba41a9fece0559748d6b6bababd2f2a5f4032adc24a555f48c5c51" Nov 04 01:30:27.609328 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.609309 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c" Nov 04 01:30:27.611546 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.611519 7059 generic.go:356] "Generic (PLEG): container finished" podID="8d6194a6-04a4-4d8d-8017-de919dc36fa5" containerID="89d5d9573cb883da8e2894ebc53363f63ee8d469a14dbfbdc8e7a1c3ac69ab71" exitCode=0 Nov 04 01:30:27.611712 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.611691 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" Nov 04 01:30:27.612319 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.611692 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" event={"ID":"8d6194a6-04a4-4d8d-8017-de919dc36fa5","Type":"ContainerDied","Data":"89d5d9573cb883da8e2894ebc53363f63ee8d469a14dbfbdc8e7a1c3ac69ab71"} Nov 04 01:30:27.612431 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.612358 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp" event={"ID":"8d6194a6-04a4-4d8d-8017-de919dc36fa5","Type":"ContainerDied","Data":"c07916e78a9542955eb54eafde118f4533966521a9ba1285c465e262e5a6905d"} Nov 04 01:30:27.614055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.614028 7059 generic.go:356] "Generic (PLEG): container finished" podID="60fe9f56-ebb0-4b4c-b325-8f3cb73c029c" containerID="54b2d67c46b6afe96124bc1cca8e4ad7a996323280e531b3967a482cba02bdd1" exitCode=0 Nov 04 01:30:27.614157 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.614090 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" event={"ID":"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c","Type":"ContainerDied","Data":"54b2d67c46b6afe96124bc1cca8e4ad7a996323280e531b3967a482cba02bdd1"} Nov 04 01:30:27.614157 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.614131 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" event={"ID":"60fe9f56-ebb0-4b4c-b325-8f3cb73c029c","Type":"ContainerDied","Data":"9ea974a1aa007a2033b0a205a4edd299886841e1396282bf13c0478a4624410d"} Nov 04 01:30:27.614255 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.614207 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62" Nov 04 01:30:27.632384 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.632365 7059 scope.go:122] "RemoveContainer" containerID="8839dc00f2ba41a9fece0559748d6b6bababd2f2a5f4032adc24a555f48c5c51" Nov 04 01:30:27.632695 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:30:27.632678 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8839dc00f2ba41a9fece0559748d6b6bababd2f2a5f4032adc24a555f48c5c51\": container with ID starting with 8839dc00f2ba41a9fece0559748d6b6bababd2f2a5f4032adc24a555f48c5c51 not found: ID does not exist" containerID="8839dc00f2ba41a9fece0559748d6b6bababd2f2a5f4032adc24a555f48c5c51" Nov 04 01:30:27.632748 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.632704 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8839dc00f2ba41a9fece0559748d6b6bababd2f2a5f4032adc24a555f48c5c51"} err="failed to get container status \"8839dc00f2ba41a9fece0559748d6b6bababd2f2a5f4032adc24a555f48c5c51\": rpc error: code = NotFound desc = could not find container \"8839dc00f2ba41a9fece0559748d6b6bababd2f2a5f4032adc24a555f48c5c51\": container with ID starting with 8839dc00f2ba41a9fece0559748d6b6bababd2f2a5f4032adc24a555f48c5c51 not found: ID does not exist" Nov 04 01:30:27.632748 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.632720 7059 scope.go:122] "RemoveContainer" containerID="89d5d9573cb883da8e2894ebc53363f63ee8d469a14dbfbdc8e7a1c3ac69ab71" Nov 04 01:30:27.643413 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.643390 7059 scope.go:122] "RemoveContainer" containerID="89d5d9573cb883da8e2894ebc53363f63ee8d469a14dbfbdc8e7a1c3ac69ab71" Nov 04 01:30:27.643707 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:30:27.643685 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89d5d9573cb883da8e2894ebc53363f63ee8d469a14dbfbdc8e7a1c3ac69ab71\": container with ID starting with 89d5d9573cb883da8e2894ebc53363f63ee8d469a14dbfbdc8e7a1c3ac69ab71 not found: ID does not exist" containerID="89d5d9573cb883da8e2894ebc53363f63ee8d469a14dbfbdc8e7a1c3ac69ab71" Nov 04 01:30:27.643800 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.643717 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89d5d9573cb883da8e2894ebc53363f63ee8d469a14dbfbdc8e7a1c3ac69ab71"} err="failed to get container status \"89d5d9573cb883da8e2894ebc53363f63ee8d469a14dbfbdc8e7a1c3ac69ab71\": rpc error: code = NotFound desc = could not find container \"89d5d9573cb883da8e2894ebc53363f63ee8d469a14dbfbdc8e7a1c3ac69ab71\": container with ID starting with 89d5d9573cb883da8e2894ebc53363f63ee8d469a14dbfbdc8e7a1c3ac69ab71 not found: ID does not exist" Nov 04 01:30:27.643800 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.643734 7059 scope.go:122] "RemoveContainer" containerID="54b2d67c46b6afe96124bc1cca8e4ad7a996323280e531b3967a482cba02bdd1" Nov 04 01:30:27.655513 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.655488 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp"] Nov 04 01:30:27.658857 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.658838 7059 scope.go:122] "RemoveContainer" containerID="54b2d67c46b6afe96124bc1cca8e4ad7a996323280e531b3967a482cba02bdd1" Nov 04 01:30:27.659283 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:30:27.659256 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54b2d67c46b6afe96124bc1cca8e4ad7a996323280e531b3967a482cba02bdd1\": container with ID starting with 54b2d67c46b6afe96124bc1cca8e4ad7a996323280e531b3967a482cba02bdd1 not found: ID does not exist" containerID="54b2d67c46b6afe96124bc1cca8e4ad7a996323280e531b3967a482cba02bdd1" Nov 04 01:30:27.659364 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.659292 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54b2d67c46b6afe96124bc1cca8e4ad7a996323280e531b3967a482cba02bdd1"} err="failed to get container status \"54b2d67c46b6afe96124bc1cca8e4ad7a996323280e531b3967a482cba02bdd1\": rpc error: code = NotFound desc = could not find container \"54b2d67c46b6afe96124bc1cca8e4ad7a996323280e531b3967a482cba02bdd1\": container with ID starting with 54b2d67c46b6afe96124bc1cca8e4ad7a996323280e531b3967a482cba02bdd1 not found: ID does not exist" Nov 04 01:30:27.660827 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.660809 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-rvfsp"] Nov 04 01:30:27.672413 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.672392 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c"] Nov 04 01:30:27.681742 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.681715 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-gf42c"] Nov 04 01:30:27.688162 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.688126 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.688162 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.688162 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.688162 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.688173 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxnrr\" (UniqueName: \"kubernetes.io/projected/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-kube-api-access-sxnrr\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.688391 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.688184 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twbdx\" (UniqueName: \"kubernetes.io/projected/8d6194a6-04a4-4d8d-8017-de919dc36fa5-kube-api-access-twbdx\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.688391 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.688193 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.688391 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.688201 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.688391 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.688209 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.688391 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.688217 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d6194a6-04a4-4d8d-8017-de919dc36fa5-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.690547 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.690525 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62"] Nov 04 01:30:27.700856 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:27.700825 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-x5b62"] Nov 04 01:30:28.171032 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:28.170994 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj"] Nov 04 01:30:28.172476 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:28.172444 7059 cpu_manager.go:414] "containerMap: removing container" podUID="8d6194a6-04a4-4d8d-8017-de919dc36fa5" containerName="istio-proxy" Nov 04 01:30:28.172658 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:28.172632 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d6194a6-04a4-4d8d-8017-de919dc36fa5" containerName="istio-proxy" Nov 04 01:30:28.172842 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:28.172821 7059 cpu_manager.go:414] "containerMap: removing container" podUID="19a17fd1-42d6-473d-9568-ad6ba68ca22a" containerName="istio-proxy" Nov 04 01:30:28.172842 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:28.172840 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19a17fd1-42d6-473d-9568-ad6ba68ca22a" containerName="istio-proxy" Nov 04 01:30:28.173054 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:28.172883 7059 cpu_manager.go:414] "containerMap: removing container" podUID="60fe9f56-ebb0-4b4c-b325-8f3cb73c029c" containerName="istio-proxy" Nov 04 01:30:28.173054 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:28.172891 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60fe9f56-ebb0-4b4c-b325-8f3cb73c029c" containerName="istio-proxy" Nov 04 01:30:28.173163 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:28.173149 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d6194a6-04a4-4d8d-8017-de919dc36fa5" containerName="istio-proxy" Nov 04 01:30:28.173218 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:28.173176 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="19a17fd1-42d6-473d-9568-ad6ba68ca22a" containerName="istio-proxy" Nov 04 01:30:28.173218 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:28.173194 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="60fe9f56-ebb0-4b4c-b325-8f3cb73c029c" containerName="istio-proxy" Nov 04 01:30:28.179390 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:28.179366 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj" Nov 04 01:30:28.195143 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:28.195110 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj"] Nov 04 01:30:28.195330 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:28.195312 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-30b1a1-serving-cert\"" Nov 04 01:30:28.299367 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:28.299276 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db4nj\" (UniqueName: \"kubernetes.io/projected/b022d16d-bb35-4822-b03c-767418cec51e-kube-api-access-db4nj\") pod \"maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj\" (UID: \"b022d16d-bb35-4822-b03c-767418cec51e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj" Nov 04 01:30:28.299795 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:28.299521 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b022d16d-bb35-4822-b03c-767418cec51e-openshift-service-ca\") pod \"maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj\" (UID: \"b022d16d-bb35-4822-b03c-767418cec51e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj" Nov 04 01:30:28.299795 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:28.299624 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b022d16d-bb35-4822-b03c-767418cec51e-maas-api-tls\") pod \"maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj\" (UID: \"b022d16d-bb35-4822-b03c-767418cec51e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj" Nov 04 01:30:28.299795 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:28.299776 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b022d16d-bb35-4822-b03c-767418cec51e-metrics-certs\") pod \"maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj\" (UID: \"b022d16d-bb35-4822-b03c-767418cec51e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj" Nov 04 01:30:28.403541 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:28.403440 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b022d16d-bb35-4822-b03c-767418cec51e-openshift-service-ca\") pod \"maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj\" (UID: \"b022d16d-bb35-4822-b03c-767418cec51e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj" Nov 04 01:30:28.403790 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:28.403550 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b022d16d-bb35-4822-b03c-767418cec51e-maas-api-tls\") pod \"maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj\" (UID: \"b022d16d-bb35-4822-b03c-767418cec51e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj" Nov 04 01:30:28.403790 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:28.403760 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b022d16d-bb35-4822-b03c-767418cec51e-metrics-certs\") pod \"maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj\" (UID: \"b022d16d-bb35-4822-b03c-767418cec51e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj" Nov 04 01:30:28.403979 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:28.403857 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db4nj\" (UniqueName: \"kubernetes.io/projected/b022d16d-bb35-4822-b03c-767418cec51e-kube-api-access-db4nj\") pod \"maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj\" (UID: \"b022d16d-bb35-4822-b03c-767418cec51e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj" Nov 04 01:30:28.404429 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:28.404406 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b022d16d-bb35-4822-b03c-767418cec51e-openshift-service-ca\") pod \"maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj\" (UID: \"b022d16d-bb35-4822-b03c-767418cec51e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj" Nov 04 01:30:28.407341 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:28.407317 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b022d16d-bb35-4822-b03c-767418cec51e-maas-api-tls\") pod \"maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj\" (UID: \"b022d16d-bb35-4822-b03c-767418cec51e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj" Nov 04 01:30:28.411776 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:28.411745 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b022d16d-bb35-4822-b03c-767418cec51e-metrics-certs\") pod \"maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj\" (UID: \"b022d16d-bb35-4822-b03c-767418cec51e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj" Nov 04 01:30:28.419677 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:28.419653 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db4nj\" (UniqueName: \"kubernetes.io/projected/b022d16d-bb35-4822-b03c-767418cec51e-kube-api-access-db4nj\") pod \"maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj\" (UID: \"b022d16d-bb35-4822-b03c-767418cec51e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj" Nov 04 01:30:28.506125 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:28.506094 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj" Nov 04 01:30:28.806590 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:28.806551 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj"] Nov 04 01:30:28.814534 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:30:28.814510 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb022d16d_bb35_4822_b03c_767418cec51e.slice/crio-bdbd9bb449dbd5e97f61153aa19b54a6d927f4a1cb9bd2989d58df55c727482a WatchSource:0}: Error finding container bdbd9bb449dbd5e97f61153aa19b54a6d927f4a1cb9bd2989d58df55c727482a: Status 404 returned error can't find the container with id bdbd9bb449dbd5e97f61153aa19b54a6d927f4a1cb9bd2989d58df55c727482a Nov 04 01:30:29.204338 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:29.204298 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19a17fd1-42d6-473d-9568-ad6ba68ca22a" path="/var/lib/kubelet/pods/19a17fd1-42d6-473d-9568-ad6ba68ca22a/volumes" Nov 04 01:30:29.205428 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:29.205401 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60fe9f56-ebb0-4b4c-b325-8f3cb73c029c" path="/var/lib/kubelet/pods/60fe9f56-ebb0-4b4c-b325-8f3cb73c029c/volumes" Nov 04 01:30:29.206138 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:29.206117 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d6194a6-04a4-4d8d-8017-de919dc36fa5" path="/var/lib/kubelet/pods/8d6194a6-04a4-4d8d-8017-de919dc36fa5/volumes" Nov 04 01:30:29.757949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:29.752129 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj" event={"ID":"b022d16d-bb35-4822-b03c-767418cec51e","Type":"ContainerStarted","Data":"b647b5c6a8b6e88bdceaa9d566717d564fc92413ac008cab7dc540c040280205"} Nov 04 01:30:29.757949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:29.752268 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj" event={"ID":"b022d16d-bb35-4822-b03c-767418cec51e","Type":"ContainerStarted","Data":"bdbd9bb449dbd5e97f61153aa19b54a6d927f4a1cb9bd2989d58df55c727482a"} Nov 04 01:30:29.757949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:29.753021 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj" Nov 04 01:30:35.766366 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:35.766336 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj" Nov 04 01:30:35.797614 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:35.797559 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj" podStartSLOduration=7.309060682 podStartE2EDuration="7.797543013s" totalImagesPullingTime="488.482331ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:29.83561261 +0000 UTC m=+1914.777461515" watchObservedRunningTime="2026-11-04 01:30:35.797543013 +0000 UTC m=+1920.739391985" Nov 04 01:30:36.484206 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:30:36.484171 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:30:37.960747 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:37.960711 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k"] Nov 04 01:30:37.968263 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:37.968237 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:30:37.983949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:37.983544 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k"] Nov 04 01:30:38.099731 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.099699 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b355cb-9a0e-45fe-8928-850062b32cdb-istiod-ca-cert\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:30:38.099901 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.099752 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8b355cb-9a0e-45fe-8928-850062b32cdb-istio-token\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:30:38.099901 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.099862 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-istio-envoy\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:30:38.099901 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.099889 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47xmf\" (UniqueName: \"kubernetes.io/projected/b8b355cb-9a0e-45fe-8928-850062b32cdb-kube-api-access-47xmf\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:30:38.100078 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.100040 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-credential-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:30:38.100078 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.100063 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-workload-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:30:38.100193 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.100178 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-workload-certs\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:30:38.100284 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.100262 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-istio-data\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:30:38.100330 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.100289 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b355cb-9a0e-45fe-8928-850062b32cdb-istio-podinfo\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:30:38.202305 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.202272 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8b355cb-9a0e-45fe-8928-850062b32cdb-istio-token\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:30:38.202471 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.202325 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-istio-envoy\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:30:38.202471 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.202430 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47xmf\" (UniqueName: \"kubernetes.io/projected/b8b355cb-9a0e-45fe-8928-850062b32cdb-kube-api-access-47xmf\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:30:38.202595 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.202490 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-credential-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:30:38.202595 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.202515 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-workload-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:30:38.202708 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.202624 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-workload-certs\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:30:38.202761 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.202750 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-istio-data\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:30:38.202811 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.202788 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b355cb-9a0e-45fe-8928-850062b32cdb-istio-podinfo\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:30:38.202905 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.202873 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-credential-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:30:38.203027 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.203002 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-workload-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:30:38.203091 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.203036 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b355cb-9a0e-45fe-8928-850062b32cdb-istiod-ca-cert\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:30:38.203283 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.203259 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-istio-data\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:30:38.203395 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.203375 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-workload-certs\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:30:38.203644 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.203623 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b355cb-9a0e-45fe-8928-850062b32cdb-istiod-ca-cert\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:30:38.204762 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.204744 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-istio-envoy\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:30:38.205297 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.205279 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b355cb-9a0e-45fe-8928-850062b32cdb-istio-podinfo\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:30:38.213584 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.213532 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47xmf\" (UniqueName: \"kubernetes.io/projected/b8b355cb-9a0e-45fe-8928-850062b32cdb-kube-api-access-47xmf\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:30:38.214118 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.214097 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8b355cb-9a0e-45fe-8928-850062b32cdb-istio-token\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:30:38.295481 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.295451 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:30:38.597021 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:30:38.596985 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b355cb_9a0e_45fe_8928_850062b32cdb.slice/crio-ca8e04ce49f60773b07fa32ed07f183c31c8dc478162868b2e1483d1e830dacd WatchSource:0}: Error finding container ca8e04ce49f60773b07fa32ed07f183c31c8dc478162868b2e1483d1e830dacd: Status 404 returned error can't find the container with id ca8e04ce49f60773b07fa32ed07f183c31c8dc478162868b2e1483d1e830dacd Nov 04 01:30:38.598080 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.598057 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k"] Nov 04 01:30:38.600568 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.600534 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:30:38.600628 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.600606 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:30:38.600661 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.600637 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:30:38.836325 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.835113 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" event={"ID":"b8b355cb-9a0e-45fe-8928-850062b32cdb","Type":"ContainerStarted","Data":"e0fd804baf641c6572e9584fffc24401af9586b278d832a34551b65ef8ce7910"} Nov 04 01:30:38.836325 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.835176 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" event={"ID":"b8b355cb-9a0e-45fe-8928-850062b32cdb","Type":"ContainerStarted","Data":"ca8e04ce49f60773b07fa32ed07f183c31c8dc478162868b2e1483d1e830dacd"} Nov 04 01:30:38.877547 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:38.877446 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" podStartSLOduration=1.877428149 podStartE2EDuration="1.877428149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:38.87362848 +0000 UTC m=+1923.815477406" watchObservedRunningTime="2026-11-04 01:30:38.877428149 +0000 UTC m=+1923.819277052" Nov 04 01:30:39.296570 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:39.296540 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:30:39.298027 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:39.297999 7059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 04 01:30:39.298158 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:39.298055 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" podUID="b8b355cb-9a0e-45fe-8928-850062b32cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 04 01:30:40.296270 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:40.296234 7059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 04 01:30:40.296453 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:40.296296 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" podUID="b8b355cb-9a0e-45fe-8928-850062b32cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 04 01:30:41.296504 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:41.296463 7059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 04 01:30:41.296970 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:41.296531 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" podUID="b8b355cb-9a0e-45fe-8928-850062b32cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 04 01:30:42.296372 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:42.296337 7059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 04 01:30:42.296628 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:42.296396 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" podUID="b8b355cb-9a0e-45fe-8928-850062b32cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 04 01:30:43.296055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:43.296015 7059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 04 01:30:43.296261 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:43.296092 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" podUID="b8b355cb-9a0e-45fe-8928-850062b32cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 04 01:30:44.296235 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:44.296202 7059 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 04 01:30:44.296612 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:44.296255 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" podUID="b8b355cb-9a0e-45fe-8928-850062b32cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 04 01:30:45.334007 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:45.328216 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:30:45.334007 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:45.329801 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:30:45.334007 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:30:45.332187 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:31:06.497082 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:31:06.497044 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:31:20.533483 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:20.533441 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7"] Nov 04 01:31:20.534758 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:20.534413 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7" podUID="6203361d-87ea-4390-ac32-dfe81c7afd2a" containerName="maas-api" containerID="cri-o://02ecf1945f9473554fd38bb08601da601797db2c95cfdbcbe307503514ec950e" gracePeriod=30 Nov 04 01:31:21.135888 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:21.135856 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7" Nov 04 01:31:21.247297 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:21.247261 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6203361d-87ea-4390-ac32-dfe81c7afd2a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6203361d-87ea-4390-ac32-dfe81c7afd2a-maas-api-tls\") pod \"6203361d-87ea-4390-ac32-dfe81c7afd2a\" (UID: \"6203361d-87ea-4390-ac32-dfe81c7afd2a\") " Nov 04 01:31:21.247521 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:21.247362 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6203361d-87ea-4390-ac32-dfe81c7afd2a-kube-api-access-p6hpt\" (UniqueName: \"kubernetes.io/projected/6203361d-87ea-4390-ac32-dfe81c7afd2a-kube-api-access-p6hpt\") pod \"6203361d-87ea-4390-ac32-dfe81c7afd2a\" (UID: \"6203361d-87ea-4390-ac32-dfe81c7afd2a\") " Nov 04 01:31:21.247521 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:21.247420 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6203361d-87ea-4390-ac32-dfe81c7afd2a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6203361d-87ea-4390-ac32-dfe81c7afd2a-openshift-service-ca\") pod \"6203361d-87ea-4390-ac32-dfe81c7afd2a\" (UID: \"6203361d-87ea-4390-ac32-dfe81c7afd2a\") " Nov 04 01:31:21.247708 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:21.247656 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6203361d-87ea-4390-ac32-dfe81c7afd2a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6203361d-87ea-4390-ac32-dfe81c7afd2a-metrics-certs\") pod \"6203361d-87ea-4390-ac32-dfe81c7afd2a\" (UID: \"6203361d-87ea-4390-ac32-dfe81c7afd2a\") " Nov 04 01:31:21.247818 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:21.247791 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6203361d-87ea-4390-ac32-dfe81c7afd2a-openshift-service-ca" pod "6203361d-87ea-4390-ac32-dfe81c7afd2a" (UID: "6203361d-87ea-4390-ac32-dfe81c7afd2a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:31:21.249095 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:21.249069 7059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6203361d-87ea-4390-ac32-dfe81c7afd2a-openshift-service-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:21.250071 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:21.250044 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6203361d-87ea-4390-ac32-dfe81c7afd2a-maas-api-tls" pod "6203361d-87ea-4390-ac32-dfe81c7afd2a" (UID: "6203361d-87ea-4390-ac32-dfe81c7afd2a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:31:21.250173 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:21.250118 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6203361d-87ea-4390-ac32-dfe81c7afd2a-metrics-certs" pod "6203361d-87ea-4390-ac32-dfe81c7afd2a" (UID: "6203361d-87ea-4390-ac32-dfe81c7afd2a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:31:21.250218 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:21.250174 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6203361d-87ea-4390-ac32-dfe81c7afd2a-kube-api-access-p6hpt" pod "6203361d-87ea-4390-ac32-dfe81c7afd2a" (UID: "6203361d-87ea-4390-ac32-dfe81c7afd2a"). InnerVolumeSpecName "kube-api-access-p6hpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:21.350687 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:21.350600 7059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6203361d-87ea-4390-ac32-dfe81c7afd2a-metrics-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:21.350687 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:21.350634 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6203361d-87ea-4390-ac32-dfe81c7afd2a-maas-api-tls\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:21.350687 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:21.350648 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6hpt\" (UniqueName: \"kubernetes.io/projected/6203361d-87ea-4390-ac32-dfe81c7afd2a-kube-api-access-p6hpt\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:21.502479 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:21.502446 7059 generic.go:356] "Generic (PLEG): container finished" podID="6203361d-87ea-4390-ac32-dfe81c7afd2a" containerID="02ecf1945f9473554fd38bb08601da601797db2c95cfdbcbe307503514ec950e" exitCode=0 Nov 04 01:31:21.502677 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:21.502496 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7" Nov 04 01:31:21.502677 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:21.502526 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7" event={"ID":"6203361d-87ea-4390-ac32-dfe81c7afd2a","Type":"ContainerDied","Data":"02ecf1945f9473554fd38bb08601da601797db2c95cfdbcbe307503514ec950e"} Nov 04 01:31:21.502677 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:21.502594 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7" event={"ID":"6203361d-87ea-4390-ac32-dfe81c7afd2a","Type":"ContainerDied","Data":"9ee73ce6f75cf84fa8bc4a3f039d2d22716273c0c5c1946579ad12655e86dc55"} Nov 04 01:31:21.502677 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:21.502607 7059 scope.go:122] "RemoveContainer" containerID="02ecf1945f9473554fd38bb08601da601797db2c95cfdbcbe307503514ec950e" Nov 04 01:31:21.517152 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:21.517135 7059 scope.go:122] "RemoveContainer" containerID="02ecf1945f9473554fd38bb08601da601797db2c95cfdbcbe307503514ec950e" Nov 04 01:31:21.517486 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:31:21.517448 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02ecf1945f9473554fd38bb08601da601797db2c95cfdbcbe307503514ec950e\": container with ID starting with 02ecf1945f9473554fd38bb08601da601797db2c95cfdbcbe307503514ec950e not found: ID does not exist" containerID="02ecf1945f9473554fd38bb08601da601797db2c95cfdbcbe307503514ec950e" Nov 04 01:31:21.517586 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:21.517482 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02ecf1945f9473554fd38bb08601da601797db2c95cfdbcbe307503514ec950e"} err="failed to get container status \"02ecf1945f9473554fd38bb08601da601797db2c95cfdbcbe307503514ec950e\": rpc error: code = NotFound desc = could not find container \"02ecf1945f9473554fd38bb08601da601797db2c95cfdbcbe307503514ec950e\": container with ID starting with 02ecf1945f9473554fd38bb08601da601797db2c95cfdbcbe307503514ec950e not found: ID does not exist" Nov 04 01:31:21.533435 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:21.533360 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7"] Nov 04 01:31:21.539390 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:21.539362 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7"] Nov 04 01:31:23.182390 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:23.182356 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6203361d-87ea-4390-ac32-dfe81c7afd2a" path="/var/lib/kubelet/pods/6203361d-87ea-4390-ac32-dfe81c7afd2a/volumes" Nov 04 01:31:24.034032 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:24.033985 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-df6da4c0-75f7c4d78-2tzs7" podUID="6203361d-87ea-4390-ac32-dfe81c7afd2a" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.115:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:31:31.534509 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:31.534456 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm"] Nov 04 01:31:31.535235 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:31.534748 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" podUID="83836442-ba13-44a5-b90d-d3ba7660d2d2" containerName="istio-proxy" containerID="cri-o://061a50b2340a540fae58b082467ff0c35adaa264eb3fc001853d4d9feb41f26f" gracePeriod=30 Nov 04 01:31:31.543600 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:31.543569 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9"] Nov 04 01:31:31.543878 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:31.543830 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" podUID="43c935f3-7c45-456d-81a7-c2bd6a38e1a0" containerName="istio-proxy" containerID="cri-o://54ecb7f98862fcee26e93182e392cebec2482f930fa85e996e9409be51813db9" gracePeriod=30 Nov 04 01:31:31.551406 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:31.551378 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz"] Nov 04 01:31:31.551720 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:31.551698 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" podUID="03ee575f-2616-424f-a7ea-79b03b554a5e" containerName="istio-proxy" containerID="cri-o://7407612ad2ee4bfe3974c8d68b3c948c3e444c1d1163343c65d025c826fff2eb" gracePeriod=30 Nov 04 01:31:31.559645 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:31.559618 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz"] Nov 04 01:31:31.559879 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:31.559837 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" podUID="50745197-72c7-41bf-a41e-23a796a4d4d0" containerName="istio-proxy" containerID="cri-o://5047e2e49d57d4ea4c8e91764ab840b25f046ceb953c77802a1292f0085f1081" gracePeriod=30 Nov 04 01:31:35.167950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:35.167588 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-385061-55d99c47d4-6mktg"] Nov 04 01:31:35.171030 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:35.169055 7059 cpu_manager.go:414] "containerMap: removing container" podUID="6203361d-87ea-4390-ac32-dfe81c7afd2a" containerName="maas-api" Nov 04 01:31:35.171030 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:35.169080 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6203361d-87ea-4390-ac32-dfe81c7afd2a" containerName="maas-api" Nov 04 01:31:35.171030 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:35.169342 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="6203361d-87ea-4390-ac32-dfe81c7afd2a" containerName="maas-api" Nov 04 01:31:35.186394 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:35.186361 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-385061-55d99c47d4-6mktg" Nov 04 01:31:35.191491 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:35.191440 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-385061-serving-cert\"" Nov 04 01:31:35.203591 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:35.203565 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-385061-55d99c47d4-6mktg"] Nov 04 01:31:35.275004 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:35.274965 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb707642-c29d-4fee-9070-06e12a582b55-maas-api-tls\") pod \"maas-api-e2e-api-a-385061-55d99c47d4-6mktg\" (UID: \"fb707642-c29d-4fee-9070-06e12a582b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-385061-55d99c47d4-6mktg" Nov 04 01:31:35.275202 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:35.275046 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb707642-c29d-4fee-9070-06e12a582b55-metrics-certs\") pod \"maas-api-e2e-api-a-385061-55d99c47d4-6mktg\" (UID: \"fb707642-c29d-4fee-9070-06e12a582b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-385061-55d99c47d4-6mktg" Nov 04 01:31:35.275276 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:35.275236 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwmf2\" (UniqueName: \"kubernetes.io/projected/fb707642-c29d-4fee-9070-06e12a582b55-kube-api-access-vwmf2\") pod \"maas-api-e2e-api-a-385061-55d99c47d4-6mktg\" (UID: \"fb707642-c29d-4fee-9070-06e12a582b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-385061-55d99c47d4-6mktg" Nov 04 01:31:35.275393 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:35.275366 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb707642-c29d-4fee-9070-06e12a582b55-openshift-service-ca\") pod \"maas-api-e2e-api-a-385061-55d99c47d4-6mktg\" (UID: \"fb707642-c29d-4fee-9070-06e12a582b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-385061-55d99c47d4-6mktg" Nov 04 01:31:35.378137 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:35.378099 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb707642-c29d-4fee-9070-06e12a582b55-maas-api-tls\") pod \"maas-api-e2e-api-a-385061-55d99c47d4-6mktg\" (UID: \"fb707642-c29d-4fee-9070-06e12a582b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-385061-55d99c47d4-6mktg" Nov 04 01:31:35.378362 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:35.378178 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb707642-c29d-4fee-9070-06e12a582b55-metrics-certs\") pod \"maas-api-e2e-api-a-385061-55d99c47d4-6mktg\" (UID: \"fb707642-c29d-4fee-9070-06e12a582b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-385061-55d99c47d4-6mktg" Nov 04 01:31:35.378362 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:31:35.378277 7059 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-385061-serving-cert: secret "maas-api-e2e-api-a-385061-serving-cert" not found Nov 04 01:31:35.378483 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:31:35.378370 7059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb707642-c29d-4fee-9070-06e12a582b55-maas-api-tls podName:fb707642-c29d-4fee-9070-06e12a582b55 nodeName:}" failed. No retries permitted until 2026-11-04 01:31:35.878348095 +0000 UTC m=+1980.820196976 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fb707642-c29d-4fee-9070-06e12a582b55-maas-api-tls") pod "maas-api-e2e-api-a-385061-55d99c47d4-6mktg" (UID: "fb707642-c29d-4fee-9070-06e12a582b55") : secret "maas-api-e2e-api-a-385061-serving-cert" not found Nov 04 01:31:35.378548 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:35.378286 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwmf2\" (UniqueName: \"kubernetes.io/projected/fb707642-c29d-4fee-9070-06e12a582b55-kube-api-access-vwmf2\") pod \"maas-api-e2e-api-a-385061-55d99c47d4-6mktg\" (UID: \"fb707642-c29d-4fee-9070-06e12a582b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-385061-55d99c47d4-6mktg" Nov 04 01:31:35.378681 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:35.378648 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb707642-c29d-4fee-9070-06e12a582b55-openshift-service-ca\") pod \"maas-api-e2e-api-a-385061-55d99c47d4-6mktg\" (UID: \"fb707642-c29d-4fee-9070-06e12a582b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-385061-55d99c47d4-6mktg" Nov 04 01:31:35.379755 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:35.379558 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb707642-c29d-4fee-9070-06e12a582b55-openshift-service-ca\") pod \"maas-api-e2e-api-a-385061-55d99c47d4-6mktg\" (UID: \"fb707642-c29d-4fee-9070-06e12a582b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-385061-55d99c47d4-6mktg" Nov 04 01:31:35.382697 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:35.382675 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb707642-c29d-4fee-9070-06e12a582b55-metrics-certs\") pod \"maas-api-e2e-api-a-385061-55d99c47d4-6mktg\" (UID: \"fb707642-c29d-4fee-9070-06e12a582b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-385061-55d99c47d4-6mktg" Nov 04 01:31:35.398330 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:35.398299 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwmf2\" (UniqueName: \"kubernetes.io/projected/fb707642-c29d-4fee-9070-06e12a582b55-kube-api-access-vwmf2\") pod \"maas-api-e2e-api-a-385061-55d99c47d4-6mktg\" (UID: \"fb707642-c29d-4fee-9070-06e12a582b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-385061-55d99c47d4-6mktg" Nov 04 01:31:35.892460 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:35.892423 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb707642-c29d-4fee-9070-06e12a582b55-maas-api-tls\") pod \"maas-api-e2e-api-a-385061-55d99c47d4-6mktg\" (UID: \"fb707642-c29d-4fee-9070-06e12a582b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-385061-55d99c47d4-6mktg" Nov 04 01:31:35.892653 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:31:35.892592 7059 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-385061-serving-cert: secret "maas-api-e2e-api-a-385061-serving-cert" not found Nov 04 01:31:35.892733 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:31:35.892699 7059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb707642-c29d-4fee-9070-06e12a582b55-maas-api-tls podName:fb707642-c29d-4fee-9070-06e12a582b55 nodeName:}" failed. No retries permitted until 2026-11-04 01:31:36.892676165 +0000 UTC m=+1981.834525046 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fb707642-c29d-4fee-9070-06e12a582b55-maas-api-tls") pod "maas-api-e2e-api-a-385061-55d99c47d4-6mktg" (UID: "fb707642-c29d-4fee-9070-06e12a582b55") : secret "maas-api-e2e-api-a-385061-serving-cert" not found Nov 04 01:31:36.487346 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:31:36.487307 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:31:36.916620 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:36.916578 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb707642-c29d-4fee-9070-06e12a582b55-maas-api-tls\") pod \"maas-api-e2e-api-a-385061-55d99c47d4-6mktg\" (UID: \"fb707642-c29d-4fee-9070-06e12a582b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-385061-55d99c47d4-6mktg" Nov 04 01:31:36.936294 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:36.936233 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb707642-c29d-4fee-9070-06e12a582b55-maas-api-tls\") pod \"maas-api-e2e-api-a-385061-55d99c47d4-6mktg\" (UID: \"fb707642-c29d-4fee-9070-06e12a582b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-385061-55d99c47d4-6mktg" Nov 04 01:31:37.043124 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.042248 7059 generic.go:356] "Generic (PLEG): container finished" podID="83836442-ba13-44a5-b90d-d3ba7660d2d2" containerID="061a50b2340a540fae58b082467ff0c35adaa264eb3fc001853d4d9feb41f26f" exitCode=0 Nov 04 01:31:37.043124 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.042372 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" event={"ID":"83836442-ba13-44a5-b90d-d3ba7660d2d2","Type":"ContainerDied","Data":"061a50b2340a540fae58b082467ff0c35adaa264eb3fc001853d4d9feb41f26f"} Nov 04 01:31:37.046038 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.045995 7059 generic.go:356] "Generic (PLEG): container finished" podID="03ee575f-2616-424f-a7ea-79b03b554a5e" containerID="7407612ad2ee4bfe3974c8d68b3c948c3e444c1d1163343c65d025c826fff2eb" exitCode=0 Nov 04 01:31:37.049981 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.046159 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" event={"ID":"03ee575f-2616-424f-a7ea-79b03b554a5e","Type":"ContainerDied","Data":"7407612ad2ee4bfe3974c8d68b3c948c3e444c1d1163343c65d025c826fff2eb"} Nov 04 01:31:37.049981 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.048400 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-385061-55d99c47d4-6mktg" Nov 04 01:31:37.049981 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.048648 7059 generic.go:356] "Generic (PLEG): container finished" podID="50745197-72c7-41bf-a41e-23a796a4d4d0" containerID="5047e2e49d57d4ea4c8e91764ab840b25f046ceb953c77802a1292f0085f1081" exitCode=0 Nov 04 01:31:37.049981 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.048799 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" event={"ID":"50745197-72c7-41bf-a41e-23a796a4d4d0","Type":"ContainerDied","Data":"5047e2e49d57d4ea4c8e91764ab840b25f046ceb953c77802a1292f0085f1081"} Nov 04 01:31:37.077049 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.076867 7059 generic.go:356] "Generic (PLEG): container finished" podID="43c935f3-7c45-456d-81a7-c2bd6a38e1a0" containerID="54ecb7f98862fcee26e93182e392cebec2482f930fa85e996e9409be51813db9" exitCode=0 Nov 04 01:31:37.077049 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.076979 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" event={"ID":"43c935f3-7c45-456d-81a7-c2bd6a38e1a0","Type":"ContainerDied","Data":"54ecb7f98862fcee26e93182e392cebec2482f930fa85e996e9409be51813db9"} Nov 04 01:31:37.135758 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.135729 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:31:37.239125 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.238104 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-workload-socket\") pod \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " Nov 04 01:31:37.239125 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.238220 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istiod-ca-cert\") pod \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " Nov 04 01:31:37.239125 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.238250 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istio-podinfo\") pod \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " Nov 04 01:31:37.239125 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.238281 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istio-envoy\") pod \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " Nov 04 01:31:37.239125 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.238322 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-workload-certs\") pod \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " Nov 04 01:31:37.239125 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.238408 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-credential-socket\") pod \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " Nov 04 01:31:37.239125 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.238458 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istio-data\") pod \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " Nov 04 01:31:37.239125 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.238630 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-kube-api-access-8t949\" (UniqueName: \"kubernetes.io/projected/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-kube-api-access-8t949\") pod \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " Nov 04 01:31:37.239125 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.238764 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istiod-ca-cert" pod "43c935f3-7c45-456d-81a7-c2bd6a38e1a0" (UID: "43c935f3-7c45-456d-81a7-c2bd6a38e1a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:31:37.239125 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.238846 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istio-token\" (UniqueName: \"kubernetes.io/projected/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istio-token\") pod \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\" (UID: \"43c935f3-7c45-456d-81a7-c2bd6a38e1a0\") " Nov 04 01:31:37.240180 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.240019 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-workload-socket" pod "43c935f3-7c45-456d-81a7-c2bd6a38e1a0" (UID: "43c935f3-7c45-456d-81a7-c2bd6a38e1a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.240597 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.240323 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-workload-certs" pod "43c935f3-7c45-456d-81a7-c2bd6a38e1a0" (UID: "43c935f3-7c45-456d-81a7-c2bd6a38e1a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.240597 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.240564 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.240754 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.240607 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.240754 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.240622 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.247441 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.247403 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istio-data" pod "43c935f3-7c45-456d-81a7-c2bd6a38e1a0" (UID: "43c935f3-7c45-456d-81a7-c2bd6a38e1a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.248669 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.248634 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istio-envoy" pod "43c935f3-7c45-456d-81a7-c2bd6a38e1a0" (UID: "43c935f3-7c45-456d-81a7-c2bd6a38e1a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.248833 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.248813 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-credential-socket" pod "43c935f3-7c45-456d-81a7-c2bd6a38e1a0" (UID: "43c935f3-7c45-456d-81a7-c2bd6a38e1a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.251743 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.251705 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-kube-api-access-8t949" pod "43c935f3-7c45-456d-81a7-c2bd6a38e1a0" (UID: "43c935f3-7c45-456d-81a7-c2bd6a38e1a0"). InnerVolumeSpecName "kube-api-access-8t949". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:37.256470 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.256434 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istio-podinfo" pod "43c935f3-7c45-456d-81a7-c2bd6a38e1a0" (UID: "43c935f3-7c45-456d-81a7-c2bd6a38e1a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:31:37.256736 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.256702 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istio-token" pod "43c935f3-7c45-456d-81a7-c2bd6a38e1a0" (UID: "43c935f3-7c45-456d-81a7-c2bd6a38e1a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:37.342504 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.342477 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.342504 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.342503 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.342729 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.342514 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.342729 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.342524 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.342729 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.342532 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.342729 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.342541 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t949\" (UniqueName: \"kubernetes.io/projected/43c935f3-7c45-456d-81a7-c2bd6a38e1a0-kube-api-access-8t949\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.351118 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.351092 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:31:37.356132 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.356106 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:31:37.363569 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.363553 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:31:37.442450 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:31:37.442419 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb707642_c29d_4fee_9070_06e12a582b55.slice/crio-e624a2773c77251f2d03f09b804ca136c9ff15eef8cd92490bf466731964df39 WatchSource:0}: Error finding container e624a2773c77251f2d03f09b804ca136c9ff15eef8cd92490bf466731964df39: Status 404 returned error can't find the container with id e624a2773c77251f2d03f09b804ca136c9ff15eef8cd92490bf466731964df39 Nov 04 01:31:37.444008 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.443983 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-credential-socket\") pod \"83836442-ba13-44a5-b90d-d3ba7660d2d2\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " Nov 04 01:31:37.444111 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.444093 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50745197-72c7-41bf-a41e-23a796a4d4d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50745197-72c7-41bf-a41e-23a796a4d4d0-istiod-ca-cert\") pod \"50745197-72c7-41bf-a41e-23a796a4d4d0\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " Nov 04 01:31:37.444186 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.444147 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-istio-data\") pod \"03ee575f-2616-424f-a7ea-79b03b554a5e\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " Nov 04 01:31:37.444253 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.444193 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-workload-socket\") pod \"50745197-72c7-41bf-a41e-23a796a4d4d0\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " Nov 04 01:31:37.444253 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.444217 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-credential-socket" pod "83836442-ba13-44a5-b90d-d3ba7660d2d2" (UID: "83836442-ba13-44a5-b90d-d3ba7660d2d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.444358 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.444313 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83836442-ba13-44a5-b90d-d3ba7660d2d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83836442-ba13-44a5-b90d-d3ba7660d2d2-istiod-ca-cert\") pod \"83836442-ba13-44a5-b90d-d3ba7660d2d2\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " Nov 04 01:31:37.444358 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.444356 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-istio-data\") pod \"50745197-72c7-41bf-a41e-23a796a4d4d0\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " Nov 04 01:31:37.444494 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.444468 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-workload-socket" pod "50745197-72c7-41bf-a41e-23a796a4d4d0" (UID: "50745197-72c7-41bf-a41e-23a796a4d4d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.444494 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.444429 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83836442-ba13-44a5-b90d-d3ba7660d2d2-kube-api-access-j9m2x\" (UniqueName: \"kubernetes.io/projected/83836442-ba13-44a5-b90d-d3ba7660d2d2-kube-api-access-j9m2x\") pod \"83836442-ba13-44a5-b90d-d3ba7660d2d2\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " Nov 04 01:31:37.444681 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.444524 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-workload-certs\") pod \"83836442-ba13-44a5-b90d-d3ba7660d2d2\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " Nov 04 01:31:37.444681 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.444592 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83836442-ba13-44a5-b90d-d3ba7660d2d2-istio-token\" (UniqueName: \"kubernetes.io/projected/83836442-ba13-44a5-b90d-d3ba7660d2d2-istio-token\") pod \"83836442-ba13-44a5-b90d-d3ba7660d2d2\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " Nov 04 01:31:37.444681 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.444621 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50745197-72c7-41bf-a41e-23a796a4d4d0-istiod-ca-cert" pod "50745197-72c7-41bf-a41e-23a796a4d4d0" (UID: "50745197-72c7-41bf-a41e-23a796a4d4d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:31:37.444843 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.444692 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03ee575f-2616-424f-a7ea-79b03b554a5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ee575f-2616-424f-a7ea-79b03b554a5e-istio-podinfo\") pod \"03ee575f-2616-424f-a7ea-79b03b554a5e\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " Nov 04 01:31:37.445181 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.445153 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83836442-ba13-44a5-b90d-d3ba7660d2d2-istiod-ca-cert" pod "83836442-ba13-44a5-b90d-d3ba7660d2d2" (UID: "83836442-ba13-44a5-b90d-d3ba7660d2d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:31:37.445360 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.445334 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-istio-data" pod "03ee575f-2616-424f-a7ea-79b03b554a5e" (UID: "03ee575f-2616-424f-a7ea-79b03b554a5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.445557 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.445529 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-istio-data" pod "50745197-72c7-41bf-a41e-23a796a4d4d0" (UID: "50745197-72c7-41bf-a41e-23a796a4d4d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.445954 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.445890 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-workload-certs" pod "83836442-ba13-44a5-b90d-d3ba7660d2d2" (UID: "83836442-ba13-44a5-b90d-d3ba7660d2d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.446540 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.446522 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-workload-certs\") pod \"50745197-72c7-41bf-a41e-23a796a4d4d0\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " Nov 04 01:31:37.446801 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.446784 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03ee575f-2616-424f-a7ea-79b03b554a5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ee575f-2616-424f-a7ea-79b03b554a5e-istiod-ca-cert\") pod \"03ee575f-2616-424f-a7ea-79b03b554a5e\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " Nov 04 01:31:37.446953 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.446915 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-credential-socket\") pod \"03ee575f-2616-424f-a7ea-79b03b554a5e\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " Nov 04 01:31:37.447085 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.447070 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83836442-ba13-44a5-b90d-d3ba7660d2d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83836442-ba13-44a5-b90d-d3ba7660d2d2-istio-podinfo\") pod \"83836442-ba13-44a5-b90d-d3ba7660d2d2\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " Nov 04 01:31:37.447252 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.447237 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-istio-envoy\") pod \"03ee575f-2616-424f-a7ea-79b03b554a5e\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " Nov 04 01:31:37.447361 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.447343 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03ee575f-2616-424f-a7ea-79b03b554a5e-kube-api-access-cdt66\" (UniqueName: \"kubernetes.io/projected/03ee575f-2616-424f-a7ea-79b03b554a5e-kube-api-access-cdt66\") pod \"03ee575f-2616-424f-a7ea-79b03b554a5e\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " Nov 04 01:31:37.447437 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.447382 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50745197-72c7-41bf-a41e-23a796a4d4d0-istio-token\" (UniqueName: \"kubernetes.io/projected/50745197-72c7-41bf-a41e-23a796a4d4d0-istio-token\") pod \"50745197-72c7-41bf-a41e-23a796a4d4d0\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " Nov 04 01:31:37.447437 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.447419 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-istio-envoy\") pod \"83836442-ba13-44a5-b90d-d3ba7660d2d2\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " Nov 04 01:31:37.447540 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.447453 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03ee575f-2616-424f-a7ea-79b03b554a5e-istio-token\" (UniqueName: \"kubernetes.io/projected/03ee575f-2616-424f-a7ea-79b03b554a5e-istio-token\") pod \"03ee575f-2616-424f-a7ea-79b03b554a5e\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " Nov 04 01:31:37.447540 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.447498 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50745197-72c7-41bf-a41e-23a796a4d4d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50745197-72c7-41bf-a41e-23a796a4d4d0-istio-podinfo\") pod \"50745197-72c7-41bf-a41e-23a796a4d4d0\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " Nov 04 01:31:37.447654 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.447602 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-credential-socket" pod "03ee575f-2616-424f-a7ea-79b03b554a5e" (UID: "03ee575f-2616-424f-a7ea-79b03b554a5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.447654 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.447614 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-istio-data\") pod \"83836442-ba13-44a5-b90d-d3ba7660d2d2\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " Nov 04 01:31:37.447756 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.447650 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-workload-certs\") pod \"03ee575f-2616-424f-a7ea-79b03b554a5e\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " Nov 04 01:31:37.447756 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.447726 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-credential-socket\") pod \"50745197-72c7-41bf-a41e-23a796a4d4d0\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " Nov 04 01:31:37.447866 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.447764 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-workload-socket\") pod \"03ee575f-2616-424f-a7ea-79b03b554a5e\" (UID: \"03ee575f-2616-424f-a7ea-79b03b554a5e\") " Nov 04 01:31:37.447866 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.447787 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-workload-certs" pod "50745197-72c7-41bf-a41e-23a796a4d4d0" (UID: "50745197-72c7-41bf-a41e-23a796a4d4d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.447866 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.447835 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-workload-socket\") pod \"83836442-ba13-44a5-b90d-d3ba7660d2d2\" (UID: \"83836442-ba13-44a5-b90d-d3ba7660d2d2\") " Nov 04 01:31:37.448057 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.447872 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50745197-72c7-41bf-a41e-23a796a4d4d0-kube-api-access-fvq7k\" (UniqueName: \"kubernetes.io/projected/50745197-72c7-41bf-a41e-23a796a4d4d0-kube-api-access-fvq7k\") pod \"50745197-72c7-41bf-a41e-23a796a4d4d0\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " Nov 04 01:31:37.448057 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.448014 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-istio-envoy\") pod \"50745197-72c7-41bf-a41e-23a796a4d4d0\" (UID: \"50745197-72c7-41bf-a41e-23a796a4d4d0\") " Nov 04 01:31:37.448164 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.448072 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03ee575f-2616-424f-a7ea-79b03b554a5e-istiod-ca-cert" pod "03ee575f-2616-424f-a7ea-79b03b554a5e" (UID: "03ee575f-2616-424f-a7ea-79b03b554a5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:31:37.448785 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.448759 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03ee575f-2616-424f-a7ea-79b03b554a5e-istio-podinfo" pod "03ee575f-2616-424f-a7ea-79b03b554a5e" (UID: "03ee575f-2616-424f-a7ea-79b03b554a5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:31:37.450076 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.450051 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.450174 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.450082 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50745197-72c7-41bf-a41e-23a796a4d4d0-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.450174 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.450097 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.450174 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.450111 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.450174 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.450124 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83836442-ba13-44a5-b90d-d3ba7660d2d2-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.450174 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.450138 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.450174 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.450152 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.450174 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.450167 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ee575f-2616-424f-a7ea-79b03b554a5e-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.450596 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.450181 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.450596 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.450196 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ee575f-2616-424f-a7ea-79b03b554a5e-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.450596 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.450211 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.451015 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.450989 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-workload-socket" pod "03ee575f-2616-424f-a7ea-79b03b554a5e" (UID: "03ee575f-2616-424f-a7ea-79b03b554a5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.451230 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.451205 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-istio-data" pod "83836442-ba13-44a5-b90d-d3ba7660d2d2" (UID: "83836442-ba13-44a5-b90d-d3ba7660d2d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.453245 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.453218 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-workload-certs" pod "03ee575f-2616-424f-a7ea-79b03b554a5e" (UID: "03ee575f-2616-424f-a7ea-79b03b554a5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.453429 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.453409 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-workload-socket" pod "83836442-ba13-44a5-b90d-d3ba7660d2d2" (UID: "83836442-ba13-44a5-b90d-d3ba7660d2d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.453705 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.453681 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-credential-socket" pod "50745197-72c7-41bf-a41e-23a796a4d4d0" (UID: "50745197-72c7-41bf-a41e-23a796a4d4d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.455651 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.455609 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ee575f-2616-424f-a7ea-79b03b554a5e-kube-api-access-cdt66" pod "03ee575f-2616-424f-a7ea-79b03b554a5e" (UID: "03ee575f-2616-424f-a7ea-79b03b554a5e"). InnerVolumeSpecName "kube-api-access-cdt66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:37.455651 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.455613 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-385061-55d99c47d4-6mktg"] Nov 04 01:31:37.456964 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.456891 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50745197-72c7-41bf-a41e-23a796a4d4d0-kube-api-access-fvq7k" pod "50745197-72c7-41bf-a41e-23a796a4d4d0" (UID: "50745197-72c7-41bf-a41e-23a796a4d4d0"). InnerVolumeSpecName "kube-api-access-fvq7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:37.458216 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.457965 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-istio-envoy" pod "03ee575f-2616-424f-a7ea-79b03b554a5e" (UID: "03ee575f-2616-424f-a7ea-79b03b554a5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.459059 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.459015 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83836442-ba13-44a5-b90d-d3ba7660d2d2-istio-podinfo" pod "83836442-ba13-44a5-b90d-d3ba7660d2d2" (UID: "83836442-ba13-44a5-b90d-d3ba7660d2d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:31:37.459160 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.459106 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ee575f-2616-424f-a7ea-79b03b554a5e-istio-token" pod "03ee575f-2616-424f-a7ea-79b03b554a5e" (UID: "03ee575f-2616-424f-a7ea-79b03b554a5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:37.461160 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.461125 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83836442-ba13-44a5-b90d-d3ba7660d2d2-kube-api-access-j9m2x" pod "83836442-ba13-44a5-b90d-d3ba7660d2d2" (UID: "83836442-ba13-44a5-b90d-d3ba7660d2d2"). InnerVolumeSpecName "kube-api-access-j9m2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:37.461422 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.461395 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-istio-envoy" pod "83836442-ba13-44a5-b90d-d3ba7660d2d2" (UID: "83836442-ba13-44a5-b90d-d3ba7660d2d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.461583 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.461529 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83836442-ba13-44a5-b90d-d3ba7660d2d2-istio-token" pod "83836442-ba13-44a5-b90d-d3ba7660d2d2" (UID: "83836442-ba13-44a5-b90d-d3ba7660d2d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:37.462866 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.462828 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50745197-72c7-41bf-a41e-23a796a4d4d0-istio-token" pod "50745197-72c7-41bf-a41e-23a796a4d4d0" (UID: "50745197-72c7-41bf-a41e-23a796a4d4d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:37.463281 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.463261 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50745197-72c7-41bf-a41e-23a796a4d4d0-istio-podinfo" pod "50745197-72c7-41bf-a41e-23a796a4d4d0" (UID: "50745197-72c7-41bf-a41e-23a796a4d4d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:31:37.464157 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.464136 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-istio-envoy" pod "50745197-72c7-41bf-a41e-23a796a4d4d0" (UID: "50745197-72c7-41bf-a41e-23a796a4d4d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:37.551886 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.551859 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9m2x\" (UniqueName: \"kubernetes.io/projected/83836442-ba13-44a5-b90d-d3ba7660d2d2-kube-api-access-j9m2x\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.551886 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.551885 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83836442-ba13-44a5-b90d-d3ba7660d2d2-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.552330 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.551895 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83836442-ba13-44a5-b90d-d3ba7660d2d2-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.552330 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.551905 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.552330 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.551915 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdt66\" (UniqueName: \"kubernetes.io/projected/03ee575f-2616-424f-a7ea-79b03b554a5e-kube-api-access-cdt66\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.552330 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.551942 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50745197-72c7-41bf-a41e-23a796a4d4d0-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.552330 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.551955 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.552330 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.551969 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ee575f-2616-424f-a7ea-79b03b554a5e-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.552330 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.551978 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50745197-72c7-41bf-a41e-23a796a4d4d0-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.552330 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.551986 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.552330 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.551995 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.552330 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.552006 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.552330 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.552015 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ee575f-2616-424f-a7ea-79b03b554a5e-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.552330 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.552024 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83836442-ba13-44a5-b90d-d3ba7660d2d2-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.552330 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.552033 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvq7k\" (UniqueName: \"kubernetes.io/projected/50745197-72c7-41bf-a41e-23a796a4d4d0-kube-api-access-fvq7k\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.552330 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:37.552043 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50745197-72c7-41bf-a41e-23a796a4d4d0-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:38.092335 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:38.092299 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" event={"ID":"83836442-ba13-44a5-b90d-d3ba7660d2d2","Type":"ContainerDied","Data":"9159ad10ae66630478351d66b45c7f49c489e2863a2e370df7a9ba7d530fb83c"} Nov 04 01:31:38.092460 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:38.092378 7059 scope.go:122] "RemoveContainer" containerID="061a50b2340a540fae58b082467ff0c35adaa264eb3fc001853d4d9feb41f26f" Nov 04 01:31:38.092582 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:38.092565 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm" Nov 04 01:31:38.111025 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:38.106185 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" event={"ID":"03ee575f-2616-424f-a7ea-79b03b554a5e","Type":"ContainerDied","Data":"25cd3bbafac12c9de5648e8837040960f9abf21b39874a8f3c4c37c5d1ec6b56"} Nov 04 01:31:38.111025 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:38.106308 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz" Nov 04 01:31:38.118865 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:38.118838 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" event={"ID":"50745197-72c7-41bf-a41e-23a796a4d4d0","Type":"ContainerDied","Data":"f66f11e567ac0347d1f9d0bbeb402cb9dd25ca57c3eda6492b7096f5534984ca"} Nov 04 01:31:38.119004 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:38.118975 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz" Nov 04 01:31:38.120991 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:38.120631 7059 scope.go:122] "RemoveContainer" containerID="7407612ad2ee4bfe3974c8d68b3c948c3e444c1d1163343c65d025c826fff2eb" Nov 04 01:31:38.125623 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:38.125594 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-385061-55d99c47d4-6mktg" event={"ID":"fb707642-c29d-4fee-9070-06e12a582b55","Type":"ContainerStarted","Data":"e624a2773c77251f2d03f09b804ca136c9ff15eef8cd92490bf466731964df39"} Nov 04 01:31:38.126448 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:38.126416 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-385061-55d99c47d4-6mktg" Nov 04 01:31:38.130021 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:38.129977 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" event={"ID":"43c935f3-7c45-456d-81a7-c2bd6a38e1a0","Type":"ContainerDied","Data":"17de2a259fa153494a7c5846512de9aea3a05fc2134e3d1873a7ce6a438498bd"} Nov 04 01:31:38.130181 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:38.130160 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9" Nov 04 01:31:38.143046 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:38.143017 7059 scope.go:122] "RemoveContainer" containerID="5047e2e49d57d4ea4c8e91764ab840b25f046ceb953c77802a1292f0085f1081" Nov 04 01:31:38.156534 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:38.156504 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm"] Nov 04 01:31:38.162551 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:38.162530 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-964qm"] Nov 04 01:31:38.172518 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:38.172492 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9"] Nov 04 01:31:38.176796 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:38.176775 7059 scope.go:122] "RemoveContainer" containerID="54ecb7f98862fcee26e93182e392cebec2482f930fa85e996e9409be51813db9" Nov 04 01:31:38.177547 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:38.177517 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kl8s9"] Nov 04 01:31:38.192758 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:38.192735 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz"] Nov 04 01:31:38.197234 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:38.197212 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-7v9tz"] Nov 04 01:31:38.203431 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:38.203411 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz"] Nov 04 01:31:38.210866 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:38.210845 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-l2ljz"] Nov 04 01:31:38.226231 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:38.226189 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-385061-55d99c47d4-6mktg" podStartSLOduration=2.681042246 podStartE2EDuration="3.226178009s" totalImagesPullingTime="545.135763ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:31:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:31:38.223781096 +0000 UTC m=+1983.165630003" watchObservedRunningTime="2026-11-04 01:31:38.226178009 +0000 UTC m=+1983.168026917" Nov 04 01:31:39.145966 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:39.145893 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-385061-55d99c47d4-6mktg" event={"ID":"fb707642-c29d-4fee-9070-06e12a582b55","Type":"ContainerStarted","Data":"0b0b2d0cdf5fafa87a505fe7ec4dca18eff0d75c1eaeb0622a29f8b96deef8d2"} Nov 04 01:31:39.180313 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:39.180279 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03ee575f-2616-424f-a7ea-79b03b554a5e" path="/var/lib/kubelet/pods/03ee575f-2616-424f-a7ea-79b03b554a5e/volumes" Nov 04 01:31:39.180790 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:39.180773 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43c935f3-7c45-456d-81a7-c2bd6a38e1a0" path="/var/lib/kubelet/pods/43c935f3-7c45-456d-81a7-c2bd6a38e1a0/volumes" Nov 04 01:31:39.181276 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:39.181258 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50745197-72c7-41bf-a41e-23a796a4d4d0" path="/var/lib/kubelet/pods/50745197-72c7-41bf-a41e-23a796a4d4d0/volumes" Nov 04 01:31:39.182205 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:39.182183 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83836442-ba13-44a5-b90d-d3ba7660d2d2" path="/var/lib/kubelet/pods/83836442-ba13-44a5-b90d-d3ba7660d2d2/volumes" Nov 04 01:31:40.335901 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:40.335869 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl"] Nov 04 01:31:40.337228 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:40.337201 7059 cpu_manager.go:414] "containerMap: removing container" podUID="03ee575f-2616-424f-a7ea-79b03b554a5e" containerName="istio-proxy" Nov 04 01:31:40.337228 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:40.337228 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ee575f-2616-424f-a7ea-79b03b554a5e" containerName="istio-proxy" Nov 04 01:31:40.337403 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:40.337292 7059 cpu_manager.go:414] "containerMap: removing container" podUID="43c935f3-7c45-456d-81a7-c2bd6a38e1a0" containerName="istio-proxy" Nov 04 01:31:40.337403 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:40.337301 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43c935f3-7c45-456d-81a7-c2bd6a38e1a0" containerName="istio-proxy" Nov 04 01:31:40.337403 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:40.337313 7059 cpu_manager.go:414] "containerMap: removing container" podUID="83836442-ba13-44a5-b90d-d3ba7660d2d2" containerName="istio-proxy" Nov 04 01:31:40.337403 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:40.337321 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83836442-ba13-44a5-b90d-d3ba7660d2d2" containerName="istio-proxy" Nov 04 01:31:40.337403 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:40.337368 7059 cpu_manager.go:414] "containerMap: removing container" podUID="50745197-72c7-41bf-a41e-23a796a4d4d0" containerName="istio-proxy" Nov 04 01:31:40.337403 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:40.337376 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50745197-72c7-41bf-a41e-23a796a4d4d0" containerName="istio-proxy" Nov 04 01:31:40.337722 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:40.337553 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="50745197-72c7-41bf-a41e-23a796a4d4d0" containerName="istio-proxy" Nov 04 01:31:40.337722 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:40.337576 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ee575f-2616-424f-a7ea-79b03b554a5e" containerName="istio-proxy" Nov 04 01:31:40.337722 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:40.337595 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="43c935f3-7c45-456d-81a7-c2bd6a38e1a0" containerName="istio-proxy" Nov 04 01:31:40.337722 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:40.337605 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="83836442-ba13-44a5-b90d-d3ba7660d2d2" containerName="istio-proxy" Nov 04 01:31:40.344148 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:40.344124 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl" Nov 04 01:31:40.353752 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:40.353713 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-4f8dd0-serving-cert\"" Nov 04 01:31:40.354669 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:40.354642 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl"] Nov 04 01:31:40.419647 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:40.417544 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr6c6\" (UniqueName: \"kubernetes.io/projected/bfeb8a23-780b-4ac0-9308-afdd49e42175-kube-api-access-mr6c6\") pod \"maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl\" (UID: \"bfeb8a23-780b-4ac0-9308-afdd49e42175\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl" Nov 04 01:31:40.419647 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:40.417680 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfeb8a23-780b-4ac0-9308-afdd49e42175-openshift-service-ca\") pod \"maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl\" (UID: \"bfeb8a23-780b-4ac0-9308-afdd49e42175\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl" Nov 04 01:31:40.419647 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:40.417776 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfeb8a23-780b-4ac0-9308-afdd49e42175-metrics-certs\") pod \"maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl\" (UID: \"bfeb8a23-780b-4ac0-9308-afdd49e42175\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl" Nov 04 01:31:40.419647 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:40.417894 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfeb8a23-780b-4ac0-9308-afdd49e42175-maas-api-tls\") pod \"maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl\" (UID: \"bfeb8a23-780b-4ac0-9308-afdd49e42175\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl" Nov 04 01:31:40.520755 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:40.520702 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfeb8a23-780b-4ac0-9308-afdd49e42175-metrics-certs\") pod \"maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl\" (UID: \"bfeb8a23-780b-4ac0-9308-afdd49e42175\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl" Nov 04 01:31:40.522088 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:40.522053 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfeb8a23-780b-4ac0-9308-afdd49e42175-maas-api-tls\") pod \"maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl\" (UID: \"bfeb8a23-780b-4ac0-9308-afdd49e42175\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl" Nov 04 01:31:40.522437 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:40.522301 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr6c6\" (UniqueName: \"kubernetes.io/projected/bfeb8a23-780b-4ac0-9308-afdd49e42175-kube-api-access-mr6c6\") pod \"maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl\" (UID: \"bfeb8a23-780b-4ac0-9308-afdd49e42175\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl" Nov 04 01:31:40.522574 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:40.522496 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfeb8a23-780b-4ac0-9308-afdd49e42175-openshift-service-ca\") pod \"maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl\" (UID: \"bfeb8a23-780b-4ac0-9308-afdd49e42175\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl" Nov 04 01:31:40.523842 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:40.523816 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfeb8a23-780b-4ac0-9308-afdd49e42175-openshift-service-ca\") pod \"maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl\" (UID: \"bfeb8a23-780b-4ac0-9308-afdd49e42175\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl" Nov 04 01:31:40.533386 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:40.533288 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfeb8a23-780b-4ac0-9308-afdd49e42175-metrics-certs\") pod \"maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl\" (UID: \"bfeb8a23-780b-4ac0-9308-afdd49e42175\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl" Nov 04 01:31:40.543805 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:40.543771 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfeb8a23-780b-4ac0-9308-afdd49e42175-maas-api-tls\") pod \"maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl\" (UID: \"bfeb8a23-780b-4ac0-9308-afdd49e42175\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl" Nov 04 01:31:40.546358 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:40.546308 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr6c6\" (UniqueName: \"kubernetes.io/projected/bfeb8a23-780b-4ac0-9308-afdd49e42175-kube-api-access-mr6c6\") pod \"maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl\" (UID: \"bfeb8a23-780b-4ac0-9308-afdd49e42175\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl" Nov 04 01:31:40.689819 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:40.689790 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl" Nov 04 01:31:41.050766 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:41.050716 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl"] Nov 04 01:31:41.062745 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:31:41.062692 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfeb8a23_780b_4ac0_9308_afdd49e42175.slice/crio-f101e161ca84aeba61566ed175911775ab17f110f565f8110a4c211137aaaedb WatchSource:0}: Error finding container f101e161ca84aeba61566ed175911775ab17f110f565f8110a4c211137aaaedb: Status 404 returned error can't find the container with id f101e161ca84aeba61566ed175911775ab17f110f565f8110a4c211137aaaedb Nov 04 01:31:41.194180 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:41.194147 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl" event={"ID":"bfeb8a23-780b-4ac0-9308-afdd49e42175","Type":"ContainerStarted","Data":"f101e161ca84aeba61566ed175911775ab17f110f565f8110a4c211137aaaedb"} Nov 04 01:31:42.215277 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:42.215226 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl" event={"ID":"bfeb8a23-780b-4ac0-9308-afdd49e42175","Type":"ContainerStarted","Data":"47f07174dd08d06d4e525eb0ae4e28bec64e3a0c0c8c6241be8db278d679b1c6"} Nov 04 01:31:42.215874 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:42.215367 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl" Nov 04 01:31:42.243814 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:42.243755 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl" podStartSLOduration=1.716823625 podStartE2EDuration="2.243736504s" totalImagesPullingTime="526.912879ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:31:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:31:42.236574766 +0000 UTC m=+1987.178423713" watchObservedRunningTime="2026-11-04 01:31:42.243736504 +0000 UTC m=+1987.185585407" Nov 04 01:31:44.958380 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:44.958299 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-30b1a1-84477cd7fd-kdnpv"] Nov 04 01:31:44.973596 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:44.973567 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-30b1a1-776c9d9d59-tlcl7"] Nov 04 01:31:45.185278 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:45.185251 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-385061-55d99c47d4-6mktg" Nov 04 01:31:48.229174 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:48.229142 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl" Nov 04 01:31:49.958800 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:49.958760 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-30b1a1-84477cd7fd-kdnpv" podUID="7d7439cc-59c5-496c-8577-fb895bba5a63" containerName="payload-processing" containerID="cri-o://58350508a605077f2c3cebe1edece0591fc4832b94a3ecf38878bd3bd53c1dd7" gracePeriod=25 Nov 04 01:31:49.974334 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:49.974300 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30b1a1-776c9d9d59-tlcl7" podUID="a786b7b2-9cad-4538-844a-99fa4a135fd3" containerName="payload-pre-processing" containerID="cri-o://b37a2f851f0d365edffd79913a94f51fd114904420b8ceab7b12eb3465b22409" gracePeriod=25 Nov 04 01:31:50.375276 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:50.375250 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-30b1a1-84477cd7fd-kdnpv" Nov 04 01:31:50.379034 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:50.379012 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30b1a1-776c9d9d59-tlcl7" Nov 04 01:31:50.438450 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:50.438421 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d7439cc-59c5-496c-8577-fb895bba5a63-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d7439cc-59c5-496c-8577-fb895bba5a63-plugins-config-volume\") pod \"7d7439cc-59c5-496c-8577-fb895bba5a63\" (UID: \"7d7439cc-59c5-496c-8577-fb895bba5a63\") " Nov 04 01:31:50.438602 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:50.438488 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a786b7b2-9cad-4538-844a-99fa4a135fd3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a786b7b2-9cad-4538-844a-99fa4a135fd3-plugins-config-volume\") pod \"a786b7b2-9cad-4538-844a-99fa4a135fd3\" (UID: \"a786b7b2-9cad-4538-844a-99fa4a135fd3\") " Nov 04 01:31:50.438727 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:50.438704 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d7439cc-59c5-496c-8577-fb895bba5a63-kube-api-access-rqlt8\" (UniqueName: \"kubernetes.io/projected/7d7439cc-59c5-496c-8577-fb895bba5a63-kube-api-access-rqlt8\") pod \"7d7439cc-59c5-496c-8577-fb895bba5a63\" (UID: \"7d7439cc-59c5-496c-8577-fb895bba5a63\") " Nov 04 01:31:50.438788 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:50.438761 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a786b7b2-9cad-4538-844a-99fa4a135fd3-kube-api-access-vb4qh\" (UniqueName: \"kubernetes.io/projected/a786b7b2-9cad-4538-844a-99fa4a135fd3-kube-api-access-vb4qh\") pod \"a786b7b2-9cad-4538-844a-99fa4a135fd3\" (UID: \"a786b7b2-9cad-4538-844a-99fa4a135fd3\") " Nov 04 01:31:50.438837 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:50.438798 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a786b7b2-9cad-4538-844a-99fa4a135fd3-plugins-config-volume" pod "a786b7b2-9cad-4538-844a-99fa4a135fd3" (UID: "a786b7b2-9cad-4538-844a-99fa4a135fd3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:31:50.438898 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:50.438868 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d7439cc-59c5-496c-8577-fb895bba5a63-plugins-config-volume" pod "7d7439cc-59c5-496c-8577-fb895bba5a63" (UID: "7d7439cc-59c5-496c-8577-fb895bba5a63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:31:50.439799 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:50.439767 7059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d7439cc-59c5-496c-8577-fb895bba5a63-plugins-config-volume\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:50.439972 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:50.439802 7059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a786b7b2-9cad-4538-844a-99fa4a135fd3-plugins-config-volume\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:50.440890 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:50.440865 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d7439cc-59c5-496c-8577-fb895bba5a63-kube-api-access-rqlt8" pod "7d7439cc-59c5-496c-8577-fb895bba5a63" (UID: "7d7439cc-59c5-496c-8577-fb895bba5a63"). InnerVolumeSpecName "kube-api-access-rqlt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:50.441192 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:50.441167 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a786b7b2-9cad-4538-844a-99fa4a135fd3-kube-api-access-vb4qh" pod "a786b7b2-9cad-4538-844a-99fa4a135fd3" (UID: "a786b7b2-9cad-4538-844a-99fa4a135fd3"). InnerVolumeSpecName "kube-api-access-vb4qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:50.512150 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:50.512120 7059 generic.go:356] "Generic (PLEG): container finished" podID="a786b7b2-9cad-4538-844a-99fa4a135fd3" containerID="b37a2f851f0d365edffd79913a94f51fd114904420b8ceab7b12eb3465b22409" exitCode=0 Nov 04 01:31:50.512306 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:50.512165 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30b1a1-776c9d9d59-tlcl7" Nov 04 01:31:50.512306 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:50.512195 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30b1a1-776c9d9d59-tlcl7" event={"ID":"a786b7b2-9cad-4538-844a-99fa4a135fd3","Type":"ContainerDied","Data":"b37a2f851f0d365edffd79913a94f51fd114904420b8ceab7b12eb3465b22409"} Nov 04 01:31:50.512306 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:50.512250 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30b1a1-776c9d9d59-tlcl7" event={"ID":"a786b7b2-9cad-4538-844a-99fa4a135fd3","Type":"ContainerDied","Data":"6e354b6a2d780414d28d329f6c415855110e9833a071131fafc50291bcd56d74"} Nov 04 01:31:50.512306 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:50.512272 7059 scope.go:122] "RemoveContainer" containerID="b37a2f851f0d365edffd79913a94f51fd114904420b8ceab7b12eb3465b22409" Nov 04 01:31:50.514072 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:50.513873 7059 generic.go:356] "Generic (PLEG): container finished" podID="7d7439cc-59c5-496c-8577-fb895bba5a63" containerID="58350508a605077f2c3cebe1edece0591fc4832b94a3ecf38878bd3bd53c1dd7" exitCode=0 Nov 04 01:31:50.514072 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:50.513950 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-30b1a1-84477cd7fd-kdnpv" Nov 04 01:31:50.514072 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:50.514002 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-30b1a1-84477cd7fd-kdnpv" event={"ID":"7d7439cc-59c5-496c-8577-fb895bba5a63","Type":"ContainerDied","Data":"58350508a605077f2c3cebe1edece0591fc4832b94a3ecf38878bd3bd53c1dd7"} Nov 04 01:31:50.514072 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:50.514040 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-30b1a1-84477cd7fd-kdnpv" event={"ID":"7d7439cc-59c5-496c-8577-fb895bba5a63","Type":"ContainerDied","Data":"505d9ab96f87175221805971fdf6e5111307e9d72e73a9be72233f7aca04776d"} Nov 04 01:31:50.525500 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:50.525441 7059 scope.go:122] "RemoveContainer" containerID="b37a2f851f0d365edffd79913a94f51fd114904420b8ceab7b12eb3465b22409" Nov 04 01:31:50.525852 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:31:50.525829 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b37a2f851f0d365edffd79913a94f51fd114904420b8ceab7b12eb3465b22409\": container with ID starting with b37a2f851f0d365edffd79913a94f51fd114904420b8ceab7b12eb3465b22409 not found: ID does not exist" containerID="b37a2f851f0d365edffd79913a94f51fd114904420b8ceab7b12eb3465b22409" Nov 04 01:31:50.525997 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:50.525862 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b37a2f851f0d365edffd79913a94f51fd114904420b8ceab7b12eb3465b22409"} err="failed to get container status \"b37a2f851f0d365edffd79913a94f51fd114904420b8ceab7b12eb3465b22409\": rpc error: code = NotFound desc = could not find container \"b37a2f851f0d365edffd79913a94f51fd114904420b8ceab7b12eb3465b22409\": container with ID starting with b37a2f851f0d365edffd79913a94f51fd114904420b8ceab7b12eb3465b22409 not found: ID does not exist" Nov 04 01:31:50.525997 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:50.525885 7059 scope.go:122] "RemoveContainer" containerID="58350508a605077f2c3cebe1edece0591fc4832b94a3ecf38878bd3bd53c1dd7" Nov 04 01:31:50.537919 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:50.537900 7059 scope.go:122] "RemoveContainer" containerID="58350508a605077f2c3cebe1edece0591fc4832b94a3ecf38878bd3bd53c1dd7" Nov 04 01:31:50.538202 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:31:50.538184 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58350508a605077f2c3cebe1edece0591fc4832b94a3ecf38878bd3bd53c1dd7\": container with ID starting with 58350508a605077f2c3cebe1edece0591fc4832b94a3ecf38878bd3bd53c1dd7 not found: ID does not exist" containerID="58350508a605077f2c3cebe1edece0591fc4832b94a3ecf38878bd3bd53c1dd7" Nov 04 01:31:50.538266 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:50.538211 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58350508a605077f2c3cebe1edece0591fc4832b94a3ecf38878bd3bd53c1dd7"} err="failed to get container status \"58350508a605077f2c3cebe1edece0591fc4832b94a3ecf38878bd3bd53c1dd7\": rpc error: code = NotFound desc = could not find container \"58350508a605077f2c3cebe1edece0591fc4832b94a3ecf38878bd3bd53c1dd7\": container with ID starting with 58350508a605077f2c3cebe1edece0591fc4832b94a3ecf38878bd3bd53c1dd7 not found: ID does not exist" Nov 04 01:31:50.541434 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:50.541409 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqlt8\" (UniqueName: \"kubernetes.io/projected/7d7439cc-59c5-496c-8577-fb895bba5a63-kube-api-access-rqlt8\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:50.541527 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:50.541436 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb4qh\" (UniqueName: \"kubernetes.io/projected/a786b7b2-9cad-4538-844a-99fa4a135fd3-kube-api-access-vb4qh\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:50.544504 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:50.544484 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-30b1a1-84477cd7fd-kdnpv"] Nov 04 01:31:50.550804 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:50.550784 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-30b1a1-84477cd7fd-kdnpv"] Nov 04 01:31:50.571822 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:50.571799 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-30b1a1-776c9d9d59-tlcl7"] Nov 04 01:31:50.581433 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:50.581407 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-30b1a1-776c9d9d59-tlcl7"] Nov 04 01:31:51.184872 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:51.184837 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d7439cc-59c5-496c-8577-fb895bba5a63" path="/var/lib/kubelet/pods/7d7439cc-59c5-496c-8577-fb895bba5a63/volumes" Nov 04 01:31:51.185527 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:51.185504 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a786b7b2-9cad-4538-844a-99fa4a135fd3" path="/var/lib/kubelet/pods/a786b7b2-9cad-4538-844a-99fa4a135fd3/volumes" Nov 04 01:31:54.421753 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:54.421722 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-920689-55545db649-5zp6c"] Nov 04 01:31:54.422603 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:54.422587 7059 cpu_manager.go:414] "containerMap: removing container" podUID="a786b7b2-9cad-4538-844a-99fa4a135fd3" containerName="payload-pre-processing" Nov 04 01:31:54.422650 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:54.422604 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a786b7b2-9cad-4538-844a-99fa4a135fd3" containerName="payload-pre-processing" Nov 04 01:31:54.422650 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:54.422617 7059 cpu_manager.go:414] "containerMap: removing container" podUID="7d7439cc-59c5-496c-8577-fb895bba5a63" containerName="payload-processing" Nov 04 01:31:54.422650 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:54.422622 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d7439cc-59c5-496c-8577-fb895bba5a63" containerName="payload-processing" Nov 04 01:31:54.422788 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:54.422778 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="a786b7b2-9cad-4538-844a-99fa4a135fd3" containerName="payload-pre-processing" Nov 04 01:31:54.422823 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:54.422788 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d7439cc-59c5-496c-8577-fb895bba5a63" containerName="payload-processing" Nov 04 01:31:54.428370 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:54.428351 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-920689-55545db649-5zp6c" Nov 04 01:31:54.435836 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:54.435800 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-920689-serving-cert\"" Nov 04 01:31:54.439207 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:54.439181 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-920689-55545db649-5zp6c"] Nov 04 01:31:54.521412 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:54.521371 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9qxz\" (UniqueName: \"kubernetes.io/projected/219c9bba-a9f4-4c59-a19d-8507db760298-kube-api-access-d9qxz\") pod \"maas-api-e2e-api-a-920689-55545db649-5zp6c\" (UID: \"219c9bba-a9f4-4c59-a19d-8507db760298\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-920689-55545db649-5zp6c" Nov 04 01:31:54.521633 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:54.521567 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/219c9bba-a9f4-4c59-a19d-8507db760298-maas-api-tls\") pod \"maas-api-e2e-api-a-920689-55545db649-5zp6c\" (UID: \"219c9bba-a9f4-4c59-a19d-8507db760298\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-920689-55545db649-5zp6c" Nov 04 01:31:54.521753 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:54.521733 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/219c9bba-a9f4-4c59-a19d-8507db760298-openshift-service-ca\") pod \"maas-api-e2e-api-a-920689-55545db649-5zp6c\" (UID: \"219c9bba-a9f4-4c59-a19d-8507db760298\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-920689-55545db649-5zp6c" Nov 04 01:31:54.522018 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:54.521989 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/219c9bba-a9f4-4c59-a19d-8507db760298-metrics-certs\") pod \"maas-api-e2e-api-a-920689-55545db649-5zp6c\" (UID: \"219c9bba-a9f4-4c59-a19d-8507db760298\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-920689-55545db649-5zp6c" Nov 04 01:31:54.626267 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:54.625559 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9qxz\" (UniqueName: \"kubernetes.io/projected/219c9bba-a9f4-4c59-a19d-8507db760298-kube-api-access-d9qxz\") pod \"maas-api-e2e-api-a-920689-55545db649-5zp6c\" (UID: \"219c9bba-a9f4-4c59-a19d-8507db760298\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-920689-55545db649-5zp6c" Nov 04 01:31:54.626267 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:54.625730 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/219c9bba-a9f4-4c59-a19d-8507db760298-maas-api-tls\") pod \"maas-api-e2e-api-a-920689-55545db649-5zp6c\" (UID: \"219c9bba-a9f4-4c59-a19d-8507db760298\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-920689-55545db649-5zp6c" Nov 04 01:31:54.626267 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:31:54.625865 7059 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-920689-serving-cert: secret "maas-api-e2e-api-a-920689-serving-cert" not found Nov 04 01:31:54.626267 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:31:54.625968 7059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/219c9bba-a9f4-4c59-a19d-8507db760298-maas-api-tls podName:219c9bba-a9f4-4c59-a19d-8507db760298 nodeName:}" failed. No retries permitted until 2026-11-04 01:31:55.125905603 +0000 UTC m=+2000.067754505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/219c9bba-a9f4-4c59-a19d-8507db760298-maas-api-tls") pod "maas-api-e2e-api-a-920689-55545db649-5zp6c" (UID: "219c9bba-a9f4-4c59-a19d-8507db760298") : secret "maas-api-e2e-api-a-920689-serving-cert" not found Nov 04 01:31:54.632960 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:54.628312 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/219c9bba-a9f4-4c59-a19d-8507db760298-openshift-service-ca\") pod \"maas-api-e2e-api-a-920689-55545db649-5zp6c\" (UID: \"219c9bba-a9f4-4c59-a19d-8507db760298\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-920689-55545db649-5zp6c" Nov 04 01:31:54.632960 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:54.628518 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/219c9bba-a9f4-4c59-a19d-8507db760298-metrics-certs\") pod \"maas-api-e2e-api-a-920689-55545db649-5zp6c\" (UID: \"219c9bba-a9f4-4c59-a19d-8507db760298\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-920689-55545db649-5zp6c" Nov 04 01:31:54.632960 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:54.629768 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/219c9bba-a9f4-4c59-a19d-8507db760298-openshift-service-ca\") pod \"maas-api-e2e-api-a-920689-55545db649-5zp6c\" (UID: \"219c9bba-a9f4-4c59-a19d-8507db760298\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-920689-55545db649-5zp6c" Nov 04 01:31:54.637000 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:54.636391 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/219c9bba-a9f4-4c59-a19d-8507db760298-metrics-certs\") pod \"maas-api-e2e-api-a-920689-55545db649-5zp6c\" (UID: \"219c9bba-a9f4-4c59-a19d-8507db760298\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-920689-55545db649-5zp6c" Nov 04 01:31:54.639546 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:54.639512 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9qxz\" (UniqueName: \"kubernetes.io/projected/219c9bba-a9f4-4c59-a19d-8507db760298-kube-api-access-d9qxz\") pod \"maas-api-e2e-api-a-920689-55545db649-5zp6c\" (UID: \"219c9bba-a9f4-4c59-a19d-8507db760298\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-920689-55545db649-5zp6c" Nov 04 01:31:55.151998 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:55.145209 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/219c9bba-a9f4-4c59-a19d-8507db760298-maas-api-tls\") pod \"maas-api-e2e-api-a-920689-55545db649-5zp6c\" (UID: \"219c9bba-a9f4-4c59-a19d-8507db760298\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-920689-55545db649-5zp6c" Nov 04 01:31:55.151998 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:55.149423 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/219c9bba-a9f4-4c59-a19d-8507db760298-maas-api-tls\") pod \"maas-api-e2e-api-a-920689-55545db649-5zp6c\" (UID: \"219c9bba-a9f4-4c59-a19d-8507db760298\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-920689-55545db649-5zp6c" Nov 04 01:31:55.354320 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:55.354280 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-920689-55545db649-5zp6c" Nov 04 01:31:55.872878 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:31:55.872845 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod219c9bba_a9f4_4c59_a19d_8507db760298.slice/crio-686cc0a8f6fe796a69f3be6982436a575ba8d38ef133e02c505730186f506385 WatchSource:0}: Error finding container 686cc0a8f6fe796a69f3be6982436a575ba8d38ef133e02c505730186f506385: Status 404 returned error can't find the container with id 686cc0a8f6fe796a69f3be6982436a575ba8d38ef133e02c505730186f506385 Nov 04 01:31:55.873345 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:55.873025 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-920689-55545db649-5zp6c"] Nov 04 01:31:56.679451 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:56.679413 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-920689-55545db649-5zp6c" event={"ID":"219c9bba-a9f4-4c59-a19d-8507db760298","Type":"ContainerStarted","Data":"45c93789f645dd46b15beebb9e3c27dcca6d478cfd5e504602e3998fa04d933a"} Nov 04 01:31:56.679631 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:56.679470 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-920689-55545db649-5zp6c" event={"ID":"219c9bba-a9f4-4c59-a19d-8507db760298","Type":"ContainerStarted","Data":"686cc0a8f6fe796a69f3be6982436a575ba8d38ef133e02c505730186f506385"} Nov 04 01:31:56.680076 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:56.680051 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-920689-55545db649-5zp6c" Nov 04 01:31:56.714809 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:31:56.714756 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-920689-55545db649-5zp6c" podStartSLOduration=2.282449149 podStartE2EDuration="2.714741556s" totalImagesPullingTime="432.292407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:31:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:31:56.708709275 +0000 UTC m=+2001.650558179" watchObservedRunningTime="2026-11-04 01:31:56.714741556 +0000 UTC m=+2001.656590459" Nov 04 01:32:02.459842 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:02.459804 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj"] Nov 04 01:32:02.460414 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:02.460036 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj" podUID="b022d16d-bb35-4822-b03c-767418cec51e" containerName="maas-api" containerID="cri-o://b647b5c6a8b6e88bdceaa9d566717d564fc92413ac008cab7dc540c040280205" gracePeriod=30 Nov 04 01:32:02.702849 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:02.702815 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-920689-55545db649-5zp6c" Nov 04 01:32:03.086005 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:03.034867 7059 generic.go:356] "Generic (PLEG): container finished" podID="b022d16d-bb35-4822-b03c-767418cec51e" containerID="b647b5c6a8b6e88bdceaa9d566717d564fc92413ac008cab7dc540c040280205" exitCode=0 Nov 04 01:32:03.103687 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:03.032023 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj" event={"ID":"b022d16d-bb35-4822-b03c-767418cec51e","Type":"ContainerDied","Data":"b647b5c6a8b6e88bdceaa9d566717d564fc92413ac008cab7dc540c040280205"} Nov 04 01:32:03.104093 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:03.104060 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj" event={"ID":"b022d16d-bb35-4822-b03c-767418cec51e","Type":"ContainerDied","Data":"bdbd9bb449dbd5e97f61153aa19b54a6d927f4a1cb9bd2989d58df55c727482a"} Nov 04 01:32:03.104229 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:03.104215 7059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdbd9bb449dbd5e97f61153aa19b54a6d927f4a1cb9bd2989d58df55c727482a" Nov 04 01:32:03.174147 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:03.171735 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj" Nov 04 01:32:03.266949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:03.266254 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b022d16d-bb35-4822-b03c-767418cec51e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b022d16d-bb35-4822-b03c-767418cec51e-metrics-certs\") pod \"b022d16d-bb35-4822-b03c-767418cec51e\" (UID: \"b022d16d-bb35-4822-b03c-767418cec51e\") " Nov 04 01:32:03.267148 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:03.267022 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b022d16d-bb35-4822-b03c-767418cec51e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b022d16d-bb35-4822-b03c-767418cec51e-openshift-service-ca\") pod \"b022d16d-bb35-4822-b03c-767418cec51e\" (UID: \"b022d16d-bb35-4822-b03c-767418cec51e\") " Nov 04 01:32:03.269419 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:03.267608 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b022d16d-bb35-4822-b03c-767418cec51e-openshift-service-ca" pod "b022d16d-bb35-4822-b03c-767418cec51e" (UID: "b022d16d-bb35-4822-b03c-767418cec51e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:03.269419 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:03.267250 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b022d16d-bb35-4822-b03c-767418cec51e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b022d16d-bb35-4822-b03c-767418cec51e-maas-api-tls\") pod \"b022d16d-bb35-4822-b03c-767418cec51e\" (UID: \"b022d16d-bb35-4822-b03c-767418cec51e\") " Nov 04 01:32:03.269419 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:03.268204 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b022d16d-bb35-4822-b03c-767418cec51e-kube-api-access-db4nj\" (UniqueName: \"kubernetes.io/projected/b022d16d-bb35-4822-b03c-767418cec51e-kube-api-access-db4nj\") pod \"b022d16d-bb35-4822-b03c-767418cec51e\" (UID: \"b022d16d-bb35-4822-b03c-767418cec51e\") " Nov 04 01:32:03.276335 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:03.272544 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b022d16d-bb35-4822-b03c-767418cec51e-maas-api-tls" pod "b022d16d-bb35-4822-b03c-767418cec51e" (UID: "b022d16d-bb35-4822-b03c-767418cec51e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:03.276335 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:03.273615 7059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b022d16d-bb35-4822-b03c-767418cec51e-openshift-service-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:03.276335 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:03.273640 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b022d16d-bb35-4822-b03c-767418cec51e-maas-api-tls\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:03.315907 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:03.310880 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b022d16d-bb35-4822-b03c-767418cec51e-metrics-certs" pod "b022d16d-bb35-4822-b03c-767418cec51e" (UID: "b022d16d-bb35-4822-b03c-767418cec51e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:03.347058 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:03.346966 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b022d16d-bb35-4822-b03c-767418cec51e-kube-api-access-db4nj" pod "b022d16d-bb35-4822-b03c-767418cec51e" (UID: "b022d16d-bb35-4822-b03c-767418cec51e"). InnerVolumeSpecName "kube-api-access-db4nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:03.376290 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:03.376249 7059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b022d16d-bb35-4822-b03c-767418cec51e-metrics-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:03.376530 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:03.376508 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db4nj\" (UniqueName: \"kubernetes.io/projected/b022d16d-bb35-4822-b03c-767418cec51e-kube-api-access-db4nj\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:04.117630 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:04.117593 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj" Nov 04 01:32:04.157726 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:04.157217 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj"] Nov 04 01:32:04.167702 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:04.167662 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-30b1a1-7f67f8fbcf-mjbmj"] Nov 04 01:32:05.425959 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:05.419170 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b022d16d-bb35-4822-b03c-767418cec51e" path="/var/lib/kubelet/pods/b022d16d-bb35-4822-b03c-767418cec51e/volumes" Nov 04 01:32:06.634778 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:32:06.634734 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:32:18.138654 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.138606 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9"] Nov 04 01:32:18.140055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.140030 7059 cpu_manager.go:414] "containerMap: removing container" podUID="b022d16d-bb35-4822-b03c-767418cec51e" containerName="maas-api" Nov 04 01:32:18.140268 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.140241 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b022d16d-bb35-4822-b03c-767418cec51e" containerName="maas-api" Nov 04 01:32:18.140506 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.140487 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b022d16d-bb35-4822-b03c-767418cec51e" containerName="maas-api" Nov 04 01:32:18.146610 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.146583 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:18.196952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.193703 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9"] Nov 04 01:32:18.271978 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.268031 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3888c434-openshift-default-dockercfg-lbhwx\"" Nov 04 01:32:18.405693 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.405172 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-istio-envoy\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:18.405693 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.405306 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/356d0098-8b70-42c6-bca0-31edc6a52ce9-istio-podinfo\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:18.405693 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.405355 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/356d0098-8b70-42c6-bca0-31edc6a52ce9-istio-token\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:18.405693 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.405490 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/356d0098-8b70-42c6-bca0-31edc6a52ce9-istiod-ca-cert\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:18.405693 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.405515 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk724\" (UniqueName: \"kubernetes.io/projected/356d0098-8b70-42c6-bca0-31edc6a52ce9-kube-api-access-tk724\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:18.406142 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.405728 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-credential-socket\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:18.406142 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.405771 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-workload-certs\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:18.406142 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.405851 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-istio-data\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:18.406142 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.406095 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-workload-socket\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:18.507552 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.507520 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-credential-socket\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:18.507728 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.507561 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-workload-certs\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:18.507728 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.507675 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-istio-data\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:18.507814 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.507800 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-workload-socket\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:18.507851 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.507842 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-istio-envoy\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:18.507904 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.507863 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-workload-certs\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:18.508013 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.507907 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/356d0098-8b70-42c6-bca0-31edc6a52ce9-istio-podinfo\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:18.508013 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.507959 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/356d0098-8b70-42c6-bca0-31edc6a52ce9-istio-token\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:18.508013 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.507969 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-credential-socket\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:18.508172 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.508070 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/356d0098-8b70-42c6-bca0-31edc6a52ce9-istiod-ca-cert\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:18.508172 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.508078 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-istio-data\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:18.508172 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.508099 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk724\" (UniqueName: \"kubernetes.io/projected/356d0098-8b70-42c6-bca0-31edc6a52ce9-kube-api-access-tk724\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:18.508172 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.508140 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-workload-socket\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:18.508680 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.508659 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/356d0098-8b70-42c6-bca0-31edc6a52ce9-istiod-ca-cert\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:18.510241 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.510216 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-istio-envoy\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:18.510619 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.510601 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/356d0098-8b70-42c6-bca0-31edc6a52ce9-istio-podinfo\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:18.520359 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.520327 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/356d0098-8b70-42c6-bca0-31edc6a52ce9-istio-token\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:18.528470 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.528447 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk724\" (UniqueName: \"kubernetes.io/projected/356d0098-8b70-42c6-bca0-31edc6a52ce9-kube-api-access-tk724\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:18.770066 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:18.770032 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:19.077323 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:32:19.077293 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod356d0098_8b70_42c6_bca0_31edc6a52ce9.slice/crio-f15d1f2359147f0899fad3f70fce89d914a5b54608438c9399d37f1fe2150385 WatchSource:0}: Error finding container f15d1f2359147f0899fad3f70fce89d914a5b54608438c9399d37f1fe2150385: Status 404 returned error can't find the container with id f15d1f2359147f0899fad3f70fce89d914a5b54608438c9399d37f1fe2150385 Nov 04 01:32:19.077674 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:19.077642 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9"] Nov 04 01:32:19.080687 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:19.080649 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:32:19.080785 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:19.080722 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:32:19.080785 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:19.080758 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:32:19.501255 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:19.501211 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" event={"ID":"356d0098-8b70-42c6-bca0-31edc6a52ce9","Type":"ContainerStarted","Data":"cb3d92dd81948aecf3840b0e1fb481b54f67030b63552e0aab6cc34570169977"} Nov 04 01:32:19.501744 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:19.501279 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" event={"ID":"356d0098-8b70-42c6-bca0-31edc6a52ce9","Type":"ContainerStarted","Data":"f15d1f2359147f0899fad3f70fce89d914a5b54608438c9399d37f1fe2150385"} Nov 04 01:32:19.553855 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:19.553788 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" podStartSLOduration=1.553767286 podStartE2EDuration="1.553767286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:32:19.550466778 +0000 UTC m=+2024.492315681" watchObservedRunningTime="2026-11-04 01:32:19.553767286 +0000 UTC m=+2024.495616193" Nov 04 01:32:19.775119 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:19.775015 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:20.774370 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:20.774316 7059 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:32:20.775070 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:20.774383 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" podUID="356d0098-8b70-42c6-bca0-31edc6a52ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:32:21.771396 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:21.771358 7059 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:32:21.771681 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:21.771651 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" podUID="356d0098-8b70-42c6-bca0-31edc6a52ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:32:22.105593 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.105512 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86"] Nov 04 01:32:22.111838 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.111813 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:22.125615 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.125583 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-workload-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:22.125824 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.125805 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-istio-data\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:22.126029 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.126000 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clnh8\" (UniqueName: \"kubernetes.io/projected/c5e9addc-09bf-4281-a52d-916e6f15cde4-kube-api-access-clnh8\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:22.126303 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.126281 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-workload-certs\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:22.126552 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.126531 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-credential-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:22.126784 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.126765 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-istio-envoy\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:22.126938 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.126900 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5e9addc-09bf-4281-a52d-916e6f15cde4-istio-token\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:22.127124 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.127092 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e9addc-09bf-4281-a52d-916e6f15cde4-istio-podinfo\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:22.127289 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.127271 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e9addc-09bf-4281-a52d-916e6f15cde4-istiod-ca-cert\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:22.133581 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.133556 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86"] Nov 04 01:32:22.198434 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.198404 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-50dab0-openshift-default-dockercfg-nmfg2\"" Nov 04 01:32:22.228738 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.228701 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e9addc-09bf-4281-a52d-916e6f15cde4-istiod-ca-cert\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:22.228893 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.228783 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-workload-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:22.228893 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.228812 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-istio-data\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:22.228893 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.228856 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clnh8\" (UniqueName: \"kubernetes.io/projected/c5e9addc-09bf-4281-a52d-916e6f15cde4-kube-api-access-clnh8\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:22.229289 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.229245 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-istio-data\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:22.229416 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.229289 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-workload-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:22.229416 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.229325 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-workload-certs\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:22.229625 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.229488 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-credential-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:22.229625 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.229614 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-istio-envoy\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:22.229816 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.229643 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5e9addc-09bf-4281-a52d-916e6f15cde4-istio-token\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:22.229816 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.229672 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e9addc-09bf-4281-a52d-916e6f15cde4-istio-podinfo\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:22.230523 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.230214 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e9addc-09bf-4281-a52d-916e6f15cde4-istiod-ca-cert\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:22.230523 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.230409 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-workload-certs\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:22.230804 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.230689 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-credential-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:22.232506 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.232487 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-istio-envoy\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:22.232808 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.232790 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e9addc-09bf-4281-a52d-916e6f15cde4-istio-podinfo\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:22.265328 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.263508 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clnh8\" (UniqueName: \"kubernetes.io/projected/c5e9addc-09bf-4281-a52d-916e6f15cde4-kube-api-access-clnh8\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:22.265328 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.265206 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5e9addc-09bf-4281-a52d-916e6f15cde4-istio-token\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:22.434294 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.434262 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:22.888988 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.886834 7059 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:32:22.888988 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.886910 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" podUID="356d0098-8b70-42c6-bca0-31edc6a52ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:32:22.975161 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.975131 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86"] Nov 04 01:32:22.976065 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.976029 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:32:22.976147 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.976095 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:32:22.976147 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:22.976127 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:32:23.733600 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:23.733562 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" event={"ID":"c5e9addc-09bf-4281-a52d-916e6f15cde4","Type":"ContainerStarted","Data":"8496c19359c429a0d2c44cd518eacae70b5faa555c4c391f1cc8baa19ff45c47"} Nov 04 01:32:23.734065 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:23.733624 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" event={"ID":"c5e9addc-09bf-4281-a52d-916e6f15cde4","Type":"ContainerStarted","Data":"b6ec477bbe4c773ac7568b0998b48e1eda485e3134e8f2e58bedefb6b231fed7"} Nov 04 01:32:23.771879 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:23.771842 7059 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:32:23.772069 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:23.771970 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" podUID="356d0098-8b70-42c6-bca0-31edc6a52ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:32:24.434637 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:24.434605 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:24.436142 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:24.436108 7059 patch_prober.go:29] interesting pod/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:32:24.436293 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:24.436184 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" podUID="c5e9addc-09bf-4281-a52d-916e6f15cde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:32:24.782043 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:24.775197 7059 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:32:24.782043 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:24.775348 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" podUID="356d0098-8b70-42c6-bca0-31edc6a52ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:32:25.435160 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:25.435132 7059 patch_prober.go:29] interesting pod/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:32:25.435398 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:25.435377 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" podUID="c5e9addc-09bf-4281-a52d-916e6f15cde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:32:25.776998 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:25.776767 7059 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 04 01:32:25.776998 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:25.776845 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" podUID="356d0098-8b70-42c6-bca0-31edc6a52ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 04 01:32:26.435654 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:26.435616 7059 patch_prober.go:29] interesting pod/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:32:26.436112 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:26.435857 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" podUID="c5e9addc-09bf-4281-a52d-916e6f15cde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:32:26.798406 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:26.798169 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:26.798788 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:26.798758 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:26.804945 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:26.800269 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:26.826430 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:26.826370 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" podStartSLOduration=4.826354601 podStartE2EDuration="4.826354601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:32:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:32:23.770058665 +0000 UTC m=+2028.711907570" watchObservedRunningTime="2026-11-04 01:32:26.826354601 +0000 UTC m=+2031.768203562" Nov 04 01:32:27.434573 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:27.434532 7059 patch_prober.go:29] interesting pod/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:32:27.434760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:27.434584 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" podUID="c5e9addc-09bf-4281-a52d-916e6f15cde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:32:28.435291 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:28.435255 7059 patch_prober.go:29] interesting pod/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:32:28.435861 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:28.435829 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" podUID="c5e9addc-09bf-4281-a52d-916e6f15cde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:32:29.469135 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:29.469097 7059 patch_prober.go:29] interesting pod/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 04 01:32:29.469793 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:29.469746 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" podUID="c5e9addc-09bf-4281-a52d-916e6f15cde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 04 01:32:30.499504 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:30.499464 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:30.500061 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:30.499834 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:30.501031 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:30.501005 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:35.262631 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:35.262586 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3888c434-7c6f7d487-56x2x"] Nov 04 01:32:35.263120 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:35.262837 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3888c434-7c6f7d487-56x2x" podUID="8b6d9ff3-bd4b-464c-851e-ac16a107ddda" containerName="maas-api" containerID="cri-o://d2a813dac09aa318cb8a5ceca0c5c49672205bf6d486f9a7c928e881236851e5" gracePeriod=30 Nov 04 01:32:35.811320 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:35.811287 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3888c434-7c6f7d487-56x2x" Nov 04 01:32:35.952350 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:35.952319 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b6d9ff3-bd4b-464c-851e-ac16a107ddda-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b6d9ff3-bd4b-464c-851e-ac16a107ddda-openshift-service-ca\") pod \"8b6d9ff3-bd4b-464c-851e-ac16a107ddda\" (UID: \"8b6d9ff3-bd4b-464c-851e-ac16a107ddda\") " Nov 04 01:32:35.952539 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:35.952423 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b6d9ff3-bd4b-464c-851e-ac16a107ddda-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b6d9ff3-bd4b-464c-851e-ac16a107ddda-maas-api-tls\") pod \"8b6d9ff3-bd4b-464c-851e-ac16a107ddda\" (UID: \"8b6d9ff3-bd4b-464c-851e-ac16a107ddda\") " Nov 04 01:32:35.952615 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:35.952569 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b6d9ff3-bd4b-464c-851e-ac16a107ddda-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b6d9ff3-bd4b-464c-851e-ac16a107ddda-metrics-certs\") pod \"8b6d9ff3-bd4b-464c-851e-ac16a107ddda\" (UID: \"8b6d9ff3-bd4b-464c-851e-ac16a107ddda\") " Nov 04 01:32:35.952703 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:35.952688 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b6d9ff3-bd4b-464c-851e-ac16a107ddda-kube-api-access-j77rt\" (UniqueName: \"kubernetes.io/projected/8b6d9ff3-bd4b-464c-851e-ac16a107ddda-kube-api-access-j77rt\") pod \"8b6d9ff3-bd4b-464c-851e-ac16a107ddda\" (UID: \"8b6d9ff3-bd4b-464c-851e-ac16a107ddda\") " Nov 04 01:32:35.952763 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:35.952723 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b6d9ff3-bd4b-464c-851e-ac16a107ddda-openshift-service-ca" pod "8b6d9ff3-bd4b-464c-851e-ac16a107ddda" (UID: "8b6d9ff3-bd4b-464c-851e-ac16a107ddda"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:35.954046 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:35.954018 7059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b6d9ff3-bd4b-464c-851e-ac16a107ddda-openshift-service-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:35.954963 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:35.954887 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b6d9ff3-bd4b-464c-851e-ac16a107ddda-maas-api-tls" pod "8b6d9ff3-bd4b-464c-851e-ac16a107ddda" (UID: "8b6d9ff3-bd4b-464c-851e-ac16a107ddda"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:35.955118 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:35.955097 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b6d9ff3-bd4b-464c-851e-ac16a107ddda-metrics-certs" pod "8b6d9ff3-bd4b-464c-851e-ac16a107ddda" (UID: "8b6d9ff3-bd4b-464c-851e-ac16a107ddda"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:35.955266 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:35.955243 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b6d9ff3-bd4b-464c-851e-ac16a107ddda-kube-api-access-j77rt" pod "8b6d9ff3-bd4b-464c-851e-ac16a107ddda" (UID: "8b6d9ff3-bd4b-464c-851e-ac16a107ddda"). InnerVolumeSpecName "kube-api-access-j77rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:36.046822 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:36.044902 7059 generic.go:356] "Generic (PLEG): container finished" podID="8b6d9ff3-bd4b-464c-851e-ac16a107ddda" containerID="d2a813dac09aa318cb8a5ceca0c5c49672205bf6d486f9a7c928e881236851e5" exitCode=0 Nov 04 01:32:36.046822 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:36.045005 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3888c434-7c6f7d487-56x2x" Nov 04 01:32:36.046822 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:36.045001 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3888c434-7c6f7d487-56x2x" event={"ID":"8b6d9ff3-bd4b-464c-851e-ac16a107ddda","Type":"ContainerDied","Data":"d2a813dac09aa318cb8a5ceca0c5c49672205bf6d486f9a7c928e881236851e5"} Nov 04 01:32:36.046822 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:36.045169 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3888c434-7c6f7d487-56x2x" event={"ID":"8b6d9ff3-bd4b-464c-851e-ac16a107ddda","Type":"ContainerDied","Data":"67dc8e30b0871e163356188330c5d972db9ca8fb472c7ac14c23a5d485ba4dd9"} Nov 04 01:32:36.046822 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:36.045197 7059 scope.go:122] "RemoveContainer" containerID="d2a813dac09aa318cb8a5ceca0c5c49672205bf6d486f9a7c928e881236851e5" Nov 04 01:32:36.055676 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:36.055649 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b6d9ff3-bd4b-464c-851e-ac16a107ddda-maas-api-tls\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:36.055807 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:36.055682 7059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b6d9ff3-bd4b-464c-851e-ac16a107ddda-metrics-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:36.055807 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:36.055699 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j77rt\" (UniqueName: \"kubernetes.io/projected/8b6d9ff3-bd4b-464c-851e-ac16a107ddda-kube-api-access-j77rt\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:36.059913 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:36.059893 7059 scope.go:122] "RemoveContainer" containerID="d2a813dac09aa318cb8a5ceca0c5c49672205bf6d486f9a7c928e881236851e5" Nov 04 01:32:36.060221 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:32:36.060202 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2a813dac09aa318cb8a5ceca0c5c49672205bf6d486f9a7c928e881236851e5\": container with ID starting with d2a813dac09aa318cb8a5ceca0c5c49672205bf6d486f9a7c928e881236851e5 not found: ID does not exist" containerID="d2a813dac09aa318cb8a5ceca0c5c49672205bf6d486f9a7c928e881236851e5" Nov 04 01:32:36.060271 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:36.060232 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2a813dac09aa318cb8a5ceca0c5c49672205bf6d486f9a7c928e881236851e5"} err="failed to get container status \"d2a813dac09aa318cb8a5ceca0c5c49672205bf6d486f9a7c928e881236851e5\": rpc error: code = NotFound desc = could not find container \"d2a813dac09aa318cb8a5ceca0c5c49672205bf6d486f9a7c928e881236851e5\": container with ID starting with d2a813dac09aa318cb8a5ceca0c5c49672205bf6d486f9a7c928e881236851e5 not found: ID does not exist" Nov 04 01:32:36.076328 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:36.076294 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3888c434-7c6f7d487-56x2x"] Nov 04 01:32:36.080844 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:36.080819 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3888c434-7c6f7d487-56x2x"] Nov 04 01:32:36.483363 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:32:36.483317 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:32:37.134172 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.134137 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf"] Nov 04 01:32:37.135048 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.135030 7059 cpu_manager.go:414] "containerMap: removing container" podUID="8b6d9ff3-bd4b-464c-851e-ac16a107ddda" containerName="maas-api" Nov 04 01:32:37.135048 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.135048 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b6d9ff3-bd4b-464c-851e-ac16a107ddda" containerName="maas-api" Nov 04 01:32:37.135273 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.135262 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b6d9ff3-bd4b-464c-851e-ac16a107ddda" containerName="maas-api" Nov 04 01:32:37.141029 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.140999 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" Nov 04 01:32:37.157916 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.157891 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf"] Nov 04 01:32:37.194775 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.194744 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b6d9ff3-bd4b-464c-851e-ac16a107ddda" path="/var/lib/kubelet/pods/8b6d9ff3-bd4b-464c-851e-ac16a107ddda/volumes" Nov 04 01:32:37.277441 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.277403 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-workload-certs\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" Nov 04 01:32:37.277623 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.277473 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-credential-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" Nov 04 01:32:37.277694 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.277617 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istio-data\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" Nov 04 01:32:37.277753 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.277683 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k96d\" (UniqueName: \"kubernetes.io/projected/19fae239-0a50-4af6-be9b-e56c6d1bdb47-kube-api-access-4k96d\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" Nov 04 01:32:37.277753 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.277740 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istio-envoy\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" Nov 04 01:32:37.277899 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.277877 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istio-podinfo\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" Nov 04 01:32:37.278008 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.277994 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istiod-ca-cert\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" Nov 04 01:32:37.278121 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.278104 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istio-token\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" Nov 04 01:32:37.278187 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.278145 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-workload-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" Nov 04 01:32:37.379803 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.379769 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istio-token\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" Nov 04 01:32:37.380105 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.379817 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-workload-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" Nov 04 01:32:37.380105 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.379902 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-workload-certs\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" Nov 04 01:32:37.380105 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.380025 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-credential-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" Nov 04 01:32:37.380269 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.380113 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istio-data\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" Nov 04 01:32:37.380269 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.380155 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k96d\" (UniqueName: \"kubernetes.io/projected/19fae239-0a50-4af6-be9b-e56c6d1bdb47-kube-api-access-4k96d\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" Nov 04 01:32:37.380269 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.380211 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istio-envoy\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" Nov 04 01:32:37.380402 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.380273 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-workload-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" Nov 04 01:32:37.380402 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.380331 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-workload-certs\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" Nov 04 01:32:37.380402 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.380368 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istio-podinfo\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" Nov 04 01:32:37.380402 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.380387 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-credential-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" Nov 04 01:32:37.380634 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.380469 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istio-data\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" Nov 04 01:32:37.380634 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.380502 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istiod-ca-cert\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" Nov 04 01:32:37.381085 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.381060 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istiod-ca-cert\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" Nov 04 01:32:37.383445 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.383408 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istio-podinfo\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" Nov 04 01:32:37.383565 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.383480 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istio-envoy\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" Nov 04 01:32:37.389778 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.389680 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istio-token\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" Nov 04 01:32:37.390425 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.390402 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k96d\" (UniqueName: \"kubernetes.io/projected/19fae239-0a50-4af6-be9b-e56c6d1bdb47-kube-api-access-4k96d\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" Nov 04 01:32:37.465238 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.465212 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" Nov 04 01:32:37.761963 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:32:37.761896 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19fae239_0a50_4af6_be9b_e56c6d1bdb47.slice/crio-55c516a8a64459b0775d8206de10545c8f157e0260b54088685b377f6efbca15 WatchSource:0}: Error finding container 55c516a8a64459b0775d8206de10545c8f157e0260b54088685b377f6efbca15: Status 404 returned error can't find the container with id 55c516a8a64459b0775d8206de10545c8f157e0260b54088685b377f6efbca15 Nov 04 01:32:37.762297 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.762067 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf"] Nov 04 01:32:37.765297 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.765265 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:32:37.765382 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.765329 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:32:37.765382 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:37.765361 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:32:38.079001 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:38.078505 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" event={"ID":"19fae239-0a50-4af6-be9b-e56c6d1bdb47","Type":"ContainerStarted","Data":"a4487f84b0689aa436238ac6ad54e41c78cc5c91899ddd29e2ae73604e3cdb2a"} Nov 04 01:32:38.079001 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:38.078594 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" event={"ID":"19fae239-0a50-4af6-be9b-e56c6d1bdb47","Type":"ContainerStarted","Data":"55c516a8a64459b0775d8206de10545c8f157e0260b54088685b377f6efbca15"} Nov 04 01:32:38.149961 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:38.136558 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" podStartSLOduration=1.136538427 podStartE2EDuration="1.136538427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:32:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:32:38.135299162 +0000 UTC m=+2043.077148066" watchObservedRunningTime="2026-11-04 01:32:38.136538427 +0000 UTC m=+2043.078387331" Nov 04 01:32:38.466160 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:38.465991 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" Nov 04 01:32:38.469597 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:38.469554 7059 patch_prober.go:29] interesting pod/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 04 01:32:38.469765 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:38.469631 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" podUID="19fae239-0a50-4af6-be9b-e56c6d1bdb47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 04 01:32:39.005405 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:39.005364 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl"] Nov 04 01:32:39.015208 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:39.015173 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl" Nov 04 01:32:39.015979 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:39.015945 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl"] Nov 04 01:32:39.019731 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:39.019701 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-bfc54d-serving-cert\"" Nov 04 01:32:39.126225 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:39.126190 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a606a0bb-d6b9-4e97-863e-82c0666282bd-metrics-certs\") pod \"maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl\" (UID: \"a606a0bb-d6b9-4e97-863e-82c0666282bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl" Nov 04 01:32:39.126366 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:39.126357 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a606a0bb-d6b9-4e97-863e-82c0666282bd-maas-api-tls\") pod \"maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl\" (UID: \"a606a0bb-d6b9-4e97-863e-82c0666282bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl" Nov 04 01:32:39.126442 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:39.126430 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb6gk\" (UniqueName: \"kubernetes.io/projected/a606a0bb-d6b9-4e97-863e-82c0666282bd-kube-api-access-lb6gk\") pod \"maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl\" (UID: \"a606a0bb-d6b9-4e97-863e-82c0666282bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl" Nov 04 01:32:39.126481 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:39.126458 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a606a0bb-d6b9-4e97-863e-82c0666282bd-openshift-service-ca\") pod \"maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl\" (UID: \"a606a0bb-d6b9-4e97-863e-82c0666282bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl" Nov 04 01:32:39.227990 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:39.227942 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb6gk\" (UniqueName: \"kubernetes.io/projected/a606a0bb-d6b9-4e97-863e-82c0666282bd-kube-api-access-lb6gk\") pod \"maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl\" (UID: \"a606a0bb-d6b9-4e97-863e-82c0666282bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl" Nov 04 01:32:39.228234 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:39.228025 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a606a0bb-d6b9-4e97-863e-82c0666282bd-openshift-service-ca\") pod \"maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl\" (UID: \"a606a0bb-d6b9-4e97-863e-82c0666282bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl" Nov 04 01:32:39.228408 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:39.228385 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a606a0bb-d6b9-4e97-863e-82c0666282bd-metrics-certs\") pod \"maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl\" (UID: \"a606a0bb-d6b9-4e97-863e-82c0666282bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl" Nov 04 01:32:39.232415 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:39.232386 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a606a0bb-d6b9-4e97-863e-82c0666282bd-maas-api-tls\") pod \"maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl\" (UID: \"a606a0bb-d6b9-4e97-863e-82c0666282bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl" Nov 04 01:32:39.235053 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:39.229558 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a606a0bb-d6b9-4e97-863e-82c0666282bd-openshift-service-ca\") pod \"maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl\" (UID: \"a606a0bb-d6b9-4e97-863e-82c0666282bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl" Nov 04 01:32:39.235053 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:39.233318 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a606a0bb-d6b9-4e97-863e-82c0666282bd-metrics-certs\") pod \"maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl\" (UID: \"a606a0bb-d6b9-4e97-863e-82c0666282bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl" Nov 04 01:32:39.250420 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:39.246607 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb6gk\" (UniqueName: \"kubernetes.io/projected/a606a0bb-d6b9-4e97-863e-82c0666282bd-kube-api-access-lb6gk\") pod \"maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl\" (UID: \"a606a0bb-d6b9-4e97-863e-82c0666282bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl" Nov 04 01:32:39.265719 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:39.258072 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a606a0bb-d6b9-4e97-863e-82c0666282bd-maas-api-tls\") pod \"maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl\" (UID: \"a606a0bb-d6b9-4e97-863e-82c0666282bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl" Nov 04 01:32:39.338984 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:39.338874 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl" Nov 04 01:32:39.470252 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:39.469438 7059 patch_prober.go:29] interesting pod/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 04 01:32:39.470252 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:39.469505 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" podUID="19fae239-0a50-4af6-be9b-e56c6d1bdb47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 04 01:32:39.865982 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:39.838555 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl"] Nov 04 01:32:40.102576 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:40.102536 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl" event={"ID":"a606a0bb-d6b9-4e97-863e-82c0666282bd","Type":"ContainerStarted","Data":"86ed4a982c32041f331bdfe505cbc01d72bc2aac9e48ec75807444c287609e60"} Nov 04 01:32:40.468948 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:40.468539 7059 patch_prober.go:29] interesting pod/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 04 01:32:40.468948 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:40.468620 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" podUID="19fae239-0a50-4af6-be9b-e56c6d1bdb47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 04 01:32:41.113250 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:41.113156 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl" event={"ID":"a606a0bb-d6b9-4e97-863e-82c0666282bd","Type":"ContainerStarted","Data":"e5bcedbc0eb3db2e0572406443eece07fa6ed4a088a21622edcc7af9a280fa86"} Nov 04 01:32:41.113250 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:41.113232 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl" Nov 04 01:32:41.159946 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:41.158075 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl" podStartSLOduration=2.381376576 podStartE2EDuration="3.158052451s" totalImagesPullingTime="776.675875ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:32:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:32:41.136314012 +0000 UTC m=+2046.078162914" watchObservedRunningTime="2026-11-04 01:32:41.158052451 +0000 UTC m=+2046.099901356" Nov 04 01:32:41.306471 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:41.306439 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86"] Nov 04 01:32:41.307094 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:41.307049 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" podUID="c5e9addc-09bf-4281-a52d-916e6f15cde4" containerName="istio-proxy" containerID="cri-o://8496c19359c429a0d2c44cd518eacae70b5faa555c4c391f1cc8baa19ff45c47" gracePeriod=30 Nov 04 01:32:41.321254 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:41.321228 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf"] Nov 04 01:32:41.321405 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:41.321385 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" podUID="19fae239-0a50-4af6-be9b-e56c6d1bdb47" containerName="istio-proxy" containerID="cri-o://a4487f84b0689aa436238ac6ad54e41c78cc5c91899ddd29e2ae73604e3cdb2a" gracePeriod=30 Nov 04 01:32:43.923569 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:43.923529 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9"] Nov 04 01:32:43.924090 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:43.923829 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" podUID="356d0098-8b70-42c6-bca0-31edc6a52ce9" containerName="istio-proxy" containerID="cri-o://cb3d92dd81948aecf3840b0e1fb481b54f67030b63552e0aab6cc34570169977" gracePeriod=30 Nov 04 01:32:45.502162 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:45.502126 7059 patch_prober.go:29] interesting pod/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:32:45.502551 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:45.502181 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" podUID="c5e9addc-09bf-4281-a52d-916e6f15cde4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:32:46.804960 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:46.804909 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:46.871324 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:46.871292 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-workload-socket\") pod \"c5e9addc-09bf-4281-a52d-916e6f15cde4\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " Nov 04 01:32:46.871503 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:46.871387 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5e9addc-09bf-4281-a52d-916e6f15cde4-kube-api-access-clnh8\" (UniqueName: \"kubernetes.io/projected/c5e9addc-09bf-4281-a52d-916e6f15cde4-kube-api-access-clnh8\") pod \"c5e9addc-09bf-4281-a52d-916e6f15cde4\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " Nov 04 01:32:46.871503 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:46.871475 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-istio-envoy\") pod \"c5e9addc-09bf-4281-a52d-916e6f15cde4\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " Nov 04 01:32:46.871626 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:46.871551 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5e9addc-09bf-4281-a52d-916e6f15cde4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e9addc-09bf-4281-a52d-916e6f15cde4-istio-podinfo\") pod \"c5e9addc-09bf-4281-a52d-916e6f15cde4\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " Nov 04 01:32:46.871626 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:46.871600 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-workload-socket" pod "c5e9addc-09bf-4281-a52d-916e6f15cde4" (UID: "c5e9addc-09bf-4281-a52d-916e6f15cde4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:46.871745 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:46.871631 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-istio-data\") pod \"c5e9addc-09bf-4281-a52d-916e6f15cde4\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " Nov 04 01:32:46.871802 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:46.871754 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5e9addc-09bf-4281-a52d-916e6f15cde4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e9addc-09bf-4281-a52d-916e6f15cde4-istiod-ca-cert\") pod \"c5e9addc-09bf-4281-a52d-916e6f15cde4\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " Nov 04 01:32:46.871861 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:46.871831 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-credential-socket\") pod \"c5e9addc-09bf-4281-a52d-916e6f15cde4\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " Nov 04 01:32:46.872581 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:46.871976 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-workload-certs\") pod \"c5e9addc-09bf-4281-a52d-916e6f15cde4\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " Nov 04 01:32:46.872581 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:46.872181 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-istio-data" pod "c5e9addc-09bf-4281-a52d-916e6f15cde4" (UID: "c5e9addc-09bf-4281-a52d-916e6f15cde4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:46.872581 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:46.872270 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5e9addc-09bf-4281-a52d-916e6f15cde4-istio-token\" (UniqueName: \"kubernetes.io/projected/c5e9addc-09bf-4281-a52d-916e6f15cde4-istio-token\") pod \"c5e9addc-09bf-4281-a52d-916e6f15cde4\" (UID: \"c5e9addc-09bf-4281-a52d-916e6f15cde4\") " Nov 04 01:32:46.872581 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:46.872346 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-credential-socket" pod "c5e9addc-09bf-4281-a52d-916e6f15cde4" (UID: "c5e9addc-09bf-4281-a52d-916e6f15cde4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:46.872581 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:46.872246 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-workload-certs" pod "c5e9addc-09bf-4281-a52d-916e6f15cde4" (UID: "c5e9addc-09bf-4281-a52d-916e6f15cde4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:46.872581 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:46.872528 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5e9addc-09bf-4281-a52d-916e6f15cde4-istiod-ca-cert" pod "c5e9addc-09bf-4281-a52d-916e6f15cde4" (UID: "c5e9addc-09bf-4281-a52d-916e6f15cde4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:46.873205 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:46.873184 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e9addc-09bf-4281-a52d-916e6f15cde4-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:46.873302 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:46.873209 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:46.873302 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:46.873223 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:46.873302 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:46.873237 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:46.873302 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:46.873254 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:46.875159 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:46.875138 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-istio-envoy" pod "c5e9addc-09bf-4281-a52d-916e6f15cde4" (UID: "c5e9addc-09bf-4281-a52d-916e6f15cde4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:46.875367 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:46.875346 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5e9addc-09bf-4281-a52d-916e6f15cde4-istio-podinfo" pod "c5e9addc-09bf-4281-a52d-916e6f15cde4" (UID: "c5e9addc-09bf-4281-a52d-916e6f15cde4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:32:46.875558 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:46.875536 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e9addc-09bf-4281-a52d-916e6f15cde4-istio-token" pod "c5e9addc-09bf-4281-a52d-916e6f15cde4" (UID: "c5e9addc-09bf-4281-a52d-916e6f15cde4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:46.875616 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:46.875574 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e9addc-09bf-4281-a52d-916e6f15cde4-kube-api-access-clnh8" pod "c5e9addc-09bf-4281-a52d-916e6f15cde4" (UID: "c5e9addc-09bf-4281-a52d-916e6f15cde4"). InnerVolumeSpecName "kube-api-access-clnh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:46.974750 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:46.974718 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clnh8\" (UniqueName: \"kubernetes.io/projected/c5e9addc-09bf-4281-a52d-916e6f15cde4-kube-api-access-clnh8\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:46.974750 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:46.974746 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e9addc-09bf-4281-a52d-916e6f15cde4-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:46.974750 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:46.974756 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e9addc-09bf-4281-a52d-916e6f15cde4-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:46.974983 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:46.974764 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5e9addc-09bf-4281-a52d-916e6f15cde4-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:47.142415 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:47.142336 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl" Nov 04 01:32:47.338114 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:47.338077 7059 generic.go:356] "Generic (PLEG): container finished" podID="c5e9addc-09bf-4281-a52d-916e6f15cde4" containerID="8496c19359c429a0d2c44cd518eacae70b5faa555c4c391f1cc8baa19ff45c47" exitCode=0 Nov 04 01:32:47.338355 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:47.338318 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" event={"ID":"c5e9addc-09bf-4281-a52d-916e6f15cde4","Type":"ContainerDied","Data":"8496c19359c429a0d2c44cd518eacae70b5faa555c4c391f1cc8baa19ff45c47"} Nov 04 01:32:47.338462 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:47.338377 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" event={"ID":"c5e9addc-09bf-4281-a52d-916e6f15cde4","Type":"ContainerDied","Data":"b6ec477bbe4c773ac7568b0998b48e1eda485e3134e8f2e58bedefb6b231fed7"} Nov 04 01:32:47.338462 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:47.338407 7059 scope.go:122] "RemoveContainer" containerID="8496c19359c429a0d2c44cd518eacae70b5faa555c4c391f1cc8baa19ff45c47" Nov 04 01:32:47.338633 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:47.338608 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86" Nov 04 01:32:47.362395 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:47.362372 7059 scope.go:122] "RemoveContainer" containerID="8496c19359c429a0d2c44cd518eacae70b5faa555c4c391f1cc8baa19ff45c47" Nov 04 01:32:47.362893 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:32:47.362860 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8496c19359c429a0d2c44cd518eacae70b5faa555c4c391f1cc8baa19ff45c47\": container with ID starting with 8496c19359c429a0d2c44cd518eacae70b5faa555c4c391f1cc8baa19ff45c47 not found: ID does not exist" containerID="8496c19359c429a0d2c44cd518eacae70b5faa555c4c391f1cc8baa19ff45c47" Nov 04 01:32:47.363015 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:47.362905 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8496c19359c429a0d2c44cd518eacae70b5faa555c4c391f1cc8baa19ff45c47"} err="failed to get container status \"8496c19359c429a0d2c44cd518eacae70b5faa555c4c391f1cc8baa19ff45c47\": rpc error: code = NotFound desc = could not find container \"8496c19359c429a0d2c44cd518eacae70b5faa555c4c391f1cc8baa19ff45c47\": container with ID starting with 8496c19359c429a0d2c44cd518eacae70b5faa555c4c391f1cc8baa19ff45c47 not found: ID does not exist" Nov 04 01:32:47.382001 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:47.381916 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86"] Nov 04 01:32:47.390998 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:47.390967 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-jbl86"] Nov 04 01:32:49.306994 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:49.306174 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5e9addc-09bf-4281-a52d-916e6f15cde4" path="/var/lib/kubelet/pods/c5e9addc-09bf-4281-a52d-916e6f15cde4/volumes" Nov 04 01:32:49.361705 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:49.361666 7059 generic.go:356] "Generic (PLEG): container finished" podID="356d0098-8b70-42c6-bca0-31edc6a52ce9" containerID="cb3d92dd81948aecf3840b0e1fb481b54f67030b63552e0aab6cc34570169977" exitCode=0 Nov 04 01:32:49.361705 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:49.361699 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" event={"ID":"356d0098-8b70-42c6-bca0-31edc6a52ce9","Type":"ContainerDied","Data":"cb3d92dd81948aecf3840b0e1fb481b54f67030b63552e0aab6cc34570169977"} Nov 04 01:32:49.410256 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:49.410232 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:49.567451 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:49.567378 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-workload-certs\") pod \"356d0098-8b70-42c6-bca0-31edc6a52ce9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " Nov 04 01:32:49.567595 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:49.567459 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-istio-data\") pod \"356d0098-8b70-42c6-bca0-31edc6a52ce9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " Nov 04 01:32:49.567595 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:49.567507 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-credential-socket\") pod \"356d0098-8b70-42c6-bca0-31edc6a52ce9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " Nov 04 01:32:49.567595 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:49.567564 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/356d0098-8b70-42c6-bca0-31edc6a52ce9-kube-api-access-tk724\" (UniqueName: \"kubernetes.io/projected/356d0098-8b70-42c6-bca0-31edc6a52ce9-kube-api-access-tk724\") pod \"356d0098-8b70-42c6-bca0-31edc6a52ce9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " Nov 04 01:32:49.567704 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:49.567638 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-workload-certs" pod "356d0098-8b70-42c6-bca0-31edc6a52ce9" (UID: "356d0098-8b70-42c6-bca0-31edc6a52ce9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:49.567704 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:49.567684 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/356d0098-8b70-42c6-bca0-31edc6a52ce9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/356d0098-8b70-42c6-bca0-31edc6a52ce9-istiod-ca-cert\") pod \"356d0098-8b70-42c6-bca0-31edc6a52ce9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " Nov 04 01:32:49.567770 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:49.567703 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-istio-data" pod "356d0098-8b70-42c6-bca0-31edc6a52ce9" (UID: "356d0098-8b70-42c6-bca0-31edc6a52ce9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:49.567770 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:49.567739 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/356d0098-8b70-42c6-bca0-31edc6a52ce9-istio-token\" (UniqueName: \"kubernetes.io/projected/356d0098-8b70-42c6-bca0-31edc6a52ce9-istio-token\") pod \"356d0098-8b70-42c6-bca0-31edc6a52ce9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " Nov 04 01:32:49.567869 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:49.567797 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-workload-socket\") pod \"356d0098-8b70-42c6-bca0-31edc6a52ce9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " Nov 04 01:32:49.567869 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:49.567832 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-credential-socket" pod "356d0098-8b70-42c6-bca0-31edc6a52ce9" (UID: "356d0098-8b70-42c6-bca0-31edc6a52ce9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:49.568021 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:49.567871 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/356d0098-8b70-42c6-bca0-31edc6a52ce9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/356d0098-8b70-42c6-bca0-31edc6a52ce9-istio-podinfo\") pod \"356d0098-8b70-42c6-bca0-31edc6a52ce9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " Nov 04 01:32:49.568021 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:49.567906 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-istio-envoy\") pod \"356d0098-8b70-42c6-bca0-31edc6a52ce9\" (UID: \"356d0098-8b70-42c6-bca0-31edc6a52ce9\") " Nov 04 01:32:49.568219 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:49.568188 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-workload-socket" pod "356d0098-8b70-42c6-bca0-31edc6a52ce9" (UID: "356d0098-8b70-42c6-bca0-31edc6a52ce9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:49.568219 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:49.568189 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/356d0098-8b70-42c6-bca0-31edc6a52ce9-istiod-ca-cert" pod "356d0098-8b70-42c6-bca0-31edc6a52ce9" (UID: "356d0098-8b70-42c6-bca0-31edc6a52ce9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:49.568726 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:49.568687 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:49.568726 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:49.568707 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:49.568726 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:49.568717 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:49.568726 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:49.568727 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/356d0098-8b70-42c6-bca0-31edc6a52ce9-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:49.569022 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:49.568735 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:49.570089 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:49.570067 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-istio-envoy" pod "356d0098-8b70-42c6-bca0-31edc6a52ce9" (UID: "356d0098-8b70-42c6-bca0-31edc6a52ce9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:49.570304 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:49.570281 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/356d0098-8b70-42c6-bca0-31edc6a52ce9-istio-podinfo" pod "356d0098-8b70-42c6-bca0-31edc6a52ce9" (UID: "356d0098-8b70-42c6-bca0-31edc6a52ce9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:32:49.570586 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:49.570569 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/356d0098-8b70-42c6-bca0-31edc6a52ce9-istio-token" pod "356d0098-8b70-42c6-bca0-31edc6a52ce9" (UID: "356d0098-8b70-42c6-bca0-31edc6a52ce9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:49.570672 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:49.570601 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/356d0098-8b70-42c6-bca0-31edc6a52ce9-kube-api-access-tk724" pod "356d0098-8b70-42c6-bca0-31edc6a52ce9" (UID: "356d0098-8b70-42c6-bca0-31edc6a52ce9"). InnerVolumeSpecName "kube-api-access-tk724". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:49.670672 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:49.670637 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/356d0098-8b70-42c6-bca0-31edc6a52ce9-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:49.670672 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:49.670674 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/356d0098-8b70-42c6-bca0-31edc6a52ce9-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:49.670904 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:49.670691 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/356d0098-8b70-42c6-bca0-31edc6a52ce9-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:49.670904 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:49.670706 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk724\" (UniqueName: \"kubernetes.io/projected/356d0098-8b70-42c6-bca0-31edc6a52ce9-kube-api-access-tk724\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:50.386063 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.385973 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" event={"ID":"356d0098-8b70-42c6-bca0-31edc6a52ce9","Type":"ContainerDied","Data":"f15d1f2359147f0899fad3f70fce89d914a5b54608438c9399d37f1fe2150385"} Nov 04 01:32:50.386063 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.386056 7059 scope.go:122] "RemoveContainer" containerID="cb3d92dd81948aecf3840b0e1fb481b54f67030b63552e0aab6cc34570169977" Nov 04 01:32:50.386586 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.386227 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9" Nov 04 01:32:50.412040 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.411971 7059 generic.go:356] "Generic (PLEG): container finished" podID="19fae239-0a50-4af6-be9b-e56c6d1bdb47" containerID="a4487f84b0689aa436238ac6ad54e41c78cc5c91899ddd29e2ae73604e3cdb2a" exitCode=0 Nov 04 01:32:50.412186 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.412123 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" event={"ID":"19fae239-0a50-4af6-be9b-e56c6d1bdb47","Type":"ContainerDied","Data":"a4487f84b0689aa436238ac6ad54e41c78cc5c91899ddd29e2ae73604e3cdb2a"} Nov 04 01:32:50.451344 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.450520 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9"] Nov 04 01:32:50.460359 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.460323 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-q2lf9"] Nov 04 01:32:50.518495 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.518462 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" Nov 04 01:32:50.619648 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.619606 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istio-data\") pod \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " Nov 04 01:32:50.619851 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.619696 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-credential-socket\") pod \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " Nov 04 01:32:50.619851 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.619783 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-workload-certs\") pod \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " Nov 04 01:32:50.620014 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.619848 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istio-token\" (UniqueName: \"kubernetes.io/projected/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istio-token\") pod \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " Nov 04 01:32:50.620557 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.620510 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-credential-socket" pod "19fae239-0a50-4af6-be9b-e56c6d1bdb47" (UID: "19fae239-0a50-4af6-be9b-e56c6d1bdb47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:50.620832 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.620807 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istio-data" pod "19fae239-0a50-4af6-be9b-e56c6d1bdb47" (UID: "19fae239-0a50-4af6-be9b-e56c6d1bdb47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:50.621133 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.621110 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-workload-certs" pod "19fae239-0a50-4af6-be9b-e56c6d1bdb47" (UID: "19fae239-0a50-4af6-be9b-e56c6d1bdb47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:50.621585 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.621555 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istio-podinfo\") pod \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " Nov 04 01:32:50.621973 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.621737 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-workload-socket\") pod \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " Nov 04 01:32:50.621973 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.621845 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19fae239-0a50-4af6-be9b-e56c6d1bdb47-kube-api-access-4k96d\" (UniqueName: \"kubernetes.io/projected/19fae239-0a50-4af6-be9b-e56c6d1bdb47-kube-api-access-4k96d\") pod \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " Nov 04 01:32:50.621973 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.621890 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istio-envoy\") pod \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " Nov 04 01:32:50.622182 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.621982 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istiod-ca-cert\") pod \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\" (UID: \"19fae239-0a50-4af6-be9b-e56c6d1bdb47\") " Nov 04 01:32:50.622182 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.622056 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-workload-socket" pod "19fae239-0a50-4af6-be9b-e56c6d1bdb47" (UID: "19fae239-0a50-4af6-be9b-e56c6d1bdb47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:50.623084 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.623039 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:50.623084 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.623070 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:50.623276 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.623089 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:50.623276 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.623090 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istiod-ca-cert" pod "19fae239-0a50-4af6-be9b-e56c6d1bdb47" (UID: "19fae239-0a50-4af6-be9b-e56c6d1bdb47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:50.623276 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.623103 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:50.633317 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.632427 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19fae239-0a50-4af6-be9b-e56c6d1bdb47-kube-api-access-4k96d" pod "19fae239-0a50-4af6-be9b-e56c6d1bdb47" (UID: "19fae239-0a50-4af6-be9b-e56c6d1bdb47"). InnerVolumeSpecName "kube-api-access-4k96d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:50.633317 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.632828 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istio-podinfo" pod "19fae239-0a50-4af6-be9b-e56c6d1bdb47" (UID: "19fae239-0a50-4af6-be9b-e56c6d1bdb47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:32:50.637184 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.633713 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istio-envoy" pod "19fae239-0a50-4af6-be9b-e56c6d1bdb47" (UID: "19fae239-0a50-4af6-be9b-e56c6d1bdb47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:50.637184 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.634502 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istio-token" pod "19fae239-0a50-4af6-be9b-e56c6d1bdb47" (UID: "19fae239-0a50-4af6-be9b-e56c6d1bdb47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:50.725039 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.725008 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k96d\" (UniqueName: \"kubernetes.io/projected/19fae239-0a50-4af6-be9b-e56c6d1bdb47-kube-api-access-4k96d\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:50.725039 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.725037 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:50.725039 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.725046 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:50.725278 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.725055 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:50.725278 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:50.725063 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19fae239-0a50-4af6-be9b-e56c6d1bdb47-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:51.190754 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:51.190719 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="356d0098-8b70-42c6-bca0-31edc6a52ce9" path="/var/lib/kubelet/pods/356d0098-8b70-42c6-bca0-31edc6a52ce9/volumes" Nov 04 01:32:51.424674 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:51.424639 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" event={"ID":"19fae239-0a50-4af6-be9b-e56c6d1bdb47","Type":"ContainerDied","Data":"55c516a8a64459b0775d8206de10545c8f157e0260b54088685b377f6efbca15"} Nov 04 01:32:51.425224 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:51.424698 7059 scope.go:122] "RemoveContainer" containerID="a4487f84b0689aa436238ac6ad54e41c78cc5c91899ddd29e2ae73604e3cdb2a" Nov 04 01:32:51.425224 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:51.424655 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf" Nov 04 01:32:51.472026 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:51.471989 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf"] Nov 04 01:32:51.477639 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:51.477611 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-9ggxf"] Nov 04 01:32:53.181255 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:32:53.181214 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19fae239-0a50-4af6-be9b-e56c6d1bdb47" path="/var/lib/kubelet/pods/19fae239-0a50-4af6-be9b-e56c6d1bdb47/volumes" Nov 04 01:33:05.500801 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:05.499096 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc"] Nov 04 01:33:05.500801 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:05.499350 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc" podUID="10669ac8-e14a-4870-b1b5-f7331793d7a4" containerName="maas-api" containerID="cri-o://b8d9137e8aa438da6ff8d08c1cae0115d5203246e541c2d9c6074c9179ec5c12" gracePeriod=30 Nov 04 01:33:06.080163 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:06.080136 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc" Nov 04 01:33:06.137770 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:06.137740 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10669ac8-e14a-4870-b1b5-f7331793d7a4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10669ac8-e14a-4870-b1b5-f7331793d7a4-openshift-service-ca\") pod \"10669ac8-e14a-4870-b1b5-f7331793d7a4\" (UID: \"10669ac8-e14a-4870-b1b5-f7331793d7a4\") " Nov 04 01:33:06.138001 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:06.137813 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10669ac8-e14a-4870-b1b5-f7331793d7a4-kube-api-access-9wp6w\" (UniqueName: \"kubernetes.io/projected/10669ac8-e14a-4870-b1b5-f7331793d7a4-kube-api-access-9wp6w\") pod \"10669ac8-e14a-4870-b1b5-f7331793d7a4\" (UID: \"10669ac8-e14a-4870-b1b5-f7331793d7a4\") " Nov 04 01:33:06.138001 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:06.137942 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10669ac8-e14a-4870-b1b5-f7331793d7a4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/10669ac8-e14a-4870-b1b5-f7331793d7a4-metrics-certs\") pod \"10669ac8-e14a-4870-b1b5-f7331793d7a4\" (UID: \"10669ac8-e14a-4870-b1b5-f7331793d7a4\") " Nov 04 01:33:06.138209 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:06.138184 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10669ac8-e14a-4870-b1b5-f7331793d7a4-openshift-service-ca" pod "10669ac8-e14a-4870-b1b5-f7331793d7a4" (UID: "10669ac8-e14a-4870-b1b5-f7331793d7a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:06.138272 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:06.138224 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10669ac8-e14a-4870-b1b5-f7331793d7a4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10669ac8-e14a-4870-b1b5-f7331793d7a4-maas-api-tls\") pod \"10669ac8-e14a-4870-b1b5-f7331793d7a4\" (UID: \"10669ac8-e14a-4870-b1b5-f7331793d7a4\") " Nov 04 01:33:06.139090 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:06.139051 7059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10669ac8-e14a-4870-b1b5-f7331793d7a4-openshift-service-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:06.142069 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:06.142045 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10669ac8-e14a-4870-b1b5-f7331793d7a4-maas-api-tls" pod "10669ac8-e14a-4870-b1b5-f7331793d7a4" (UID: "10669ac8-e14a-4870-b1b5-f7331793d7a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:06.142162 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:06.142065 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10669ac8-e14a-4870-b1b5-f7331793d7a4-metrics-certs" pod "10669ac8-e14a-4870-b1b5-f7331793d7a4" (UID: "10669ac8-e14a-4870-b1b5-f7331793d7a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:06.142162 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:06.142101 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10669ac8-e14a-4870-b1b5-f7331793d7a4-kube-api-access-9wp6w" pod "10669ac8-e14a-4870-b1b5-f7331793d7a4" (UID: "10669ac8-e14a-4870-b1b5-f7331793d7a4"). InnerVolumeSpecName "kube-api-access-9wp6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:06.240578 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:06.240537 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wp6w\" (UniqueName: \"kubernetes.io/projected/10669ac8-e14a-4870-b1b5-f7331793d7a4-kube-api-access-9wp6w\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:06.240578 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:06.240568 7059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10669ac8-e14a-4870-b1b5-f7331793d7a4-metrics-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:06.240578 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:06.240578 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10669ac8-e14a-4870-b1b5-f7331793d7a4-maas-api-tls\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:06.482138 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:33:06.482098 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:33:06.600440 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:06.600409 7059 generic.go:356] "Generic (PLEG): container finished" podID="10669ac8-e14a-4870-b1b5-f7331793d7a4" containerID="b8d9137e8aa438da6ff8d08c1cae0115d5203246e541c2d9c6074c9179ec5c12" exitCode=0 Nov 04 01:33:06.600913 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:06.600520 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc" Nov 04 01:33:06.600913 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:06.600565 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc" event={"ID":"10669ac8-e14a-4870-b1b5-f7331793d7a4","Type":"ContainerDied","Data":"b8d9137e8aa438da6ff8d08c1cae0115d5203246e541c2d9c6074c9179ec5c12"} Nov 04 01:33:06.600913 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:06.600629 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc" event={"ID":"10669ac8-e14a-4870-b1b5-f7331793d7a4","Type":"ContainerDied","Data":"90473e4e5c24e1c8c0f504d432e2d5f5640aa747c864497c33de0bfc1c355330"} Nov 04 01:33:06.600913 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:06.600661 7059 scope.go:122] "RemoveContainer" containerID="b8d9137e8aa438da6ff8d08c1cae0115d5203246e541c2d9c6074c9179ec5c12" Nov 04 01:33:06.613642 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:06.613625 7059 scope.go:122] "RemoveContainer" containerID="b8d9137e8aa438da6ff8d08c1cae0115d5203246e541c2d9c6074c9179ec5c12" Nov 04 01:33:06.613963 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:33:06.613915 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8d9137e8aa438da6ff8d08c1cae0115d5203246e541c2d9c6074c9179ec5c12\": container with ID starting with b8d9137e8aa438da6ff8d08c1cae0115d5203246e541c2d9c6074c9179ec5c12 not found: ID does not exist" containerID="b8d9137e8aa438da6ff8d08c1cae0115d5203246e541c2d9c6074c9179ec5c12" Nov 04 01:33:06.613963 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:06.613956 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8d9137e8aa438da6ff8d08c1cae0115d5203246e541c2d9c6074c9179ec5c12"} err="failed to get container status \"b8d9137e8aa438da6ff8d08c1cae0115d5203246e541c2d9c6074c9179ec5c12\": rpc error: code = NotFound desc = could not find container \"b8d9137e8aa438da6ff8d08c1cae0115d5203246e541c2d9c6074c9179ec5c12\": container with ID starting with b8d9137e8aa438da6ff8d08c1cae0115d5203246e541c2d9c6074c9179ec5c12 not found: ID does not exist" Nov 04 01:33:06.645962 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:06.645933 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc"] Nov 04 01:33:06.650674 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:06.650653 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc"] Nov 04 01:33:07.181042 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:07.181010 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10669ac8-e14a-4870-b1b5-f7331793d7a4" path="/var/lib/kubelet/pods/10669ac8-e14a-4870-b1b5-f7331793d7a4/volumes" Nov 04 01:33:09.022197 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:09.022142 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-75f7ae-6458bbdb79-zl5mc" podUID="10669ac8-e14a-4870-b1b5-f7331793d7a4" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.116:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:16.111643 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:16.094305 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb"] Nov 04 01:33:16.111643 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:16.094569 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" podUID="612b8bca-a31e-4126-a517-ff449f55acd8" containerName="istio-proxy" containerID="cri-o://ba4e37ce5194dddbbc75cbaeecb38504bd9e8b011e116861608b683a10b3604b" gracePeriod=30 Nov 04 01:33:16.111643 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:16.104668 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k"] Nov 04 01:33:16.111643 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:16.105034 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" podUID="b8b355cb-9a0e-45fe-8928-850062b32cdb" containerName="istio-proxy" containerID="cri-o://e0fd804baf641c6572e9584fffc24401af9586b278d832a34551b65ef8ce7910" gracePeriod=30 Nov 04 01:33:16.400187 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:16.391594 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-385061-55d99c47d4-6mktg"] Nov 04 01:33:16.400187 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:16.391845 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-385061-55d99c47d4-6mktg" podUID="fb707642-c29d-4fee-9070-06e12a582b55" containerName="maas-api" containerID="cri-o://0b0b2d0cdf5fafa87a505fe7ec4dca18eff0d75c1eaeb0622a29f8b96deef8d2" gracePeriod=30 Nov 04 01:33:16.882758 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:16.882565 7059 generic.go:356] "Generic (PLEG): container finished" podID="fb707642-c29d-4fee-9070-06e12a582b55" containerID="0b0b2d0cdf5fafa87a505fe7ec4dca18eff0d75c1eaeb0622a29f8b96deef8d2" exitCode=0 Nov 04 01:33:16.882758 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:16.882684 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-385061-55d99c47d4-6mktg" event={"ID":"fb707642-c29d-4fee-9070-06e12a582b55","Type":"ContainerDied","Data":"0b0b2d0cdf5fafa87a505fe7ec4dca18eff0d75c1eaeb0622a29f8b96deef8d2"} Nov 04 01:33:16.922015 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:16.921990 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-385061-55d99c47d4-6mktg" Nov 04 01:33:17.005082 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:17.005052 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb707642-c29d-4fee-9070-06e12a582b55-kube-api-access-vwmf2\" (UniqueName: \"kubernetes.io/projected/fb707642-c29d-4fee-9070-06e12a582b55-kube-api-access-vwmf2\") pod \"fb707642-c29d-4fee-9070-06e12a582b55\" (UID: \"fb707642-c29d-4fee-9070-06e12a582b55\") " Nov 04 01:33:17.005276 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:17.005139 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb707642-c29d-4fee-9070-06e12a582b55-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb707642-c29d-4fee-9070-06e12a582b55-maas-api-tls\") pod \"fb707642-c29d-4fee-9070-06e12a582b55\" (UID: \"fb707642-c29d-4fee-9070-06e12a582b55\") " Nov 04 01:33:17.005276 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:17.005188 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb707642-c29d-4fee-9070-06e12a582b55-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb707642-c29d-4fee-9070-06e12a582b55-metrics-certs\") pod \"fb707642-c29d-4fee-9070-06e12a582b55\" (UID: \"fb707642-c29d-4fee-9070-06e12a582b55\") " Nov 04 01:33:17.005408 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:17.005305 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb707642-c29d-4fee-9070-06e12a582b55-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb707642-c29d-4fee-9070-06e12a582b55-openshift-service-ca\") pod \"fb707642-c29d-4fee-9070-06e12a582b55\" (UID: \"fb707642-c29d-4fee-9070-06e12a582b55\") " Nov 04 01:33:17.005710 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:17.005678 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb707642-c29d-4fee-9070-06e12a582b55-openshift-service-ca" pod "fb707642-c29d-4fee-9070-06e12a582b55" (UID: "fb707642-c29d-4fee-9070-06e12a582b55"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:17.006189 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:17.006169 7059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb707642-c29d-4fee-9070-06e12a582b55-openshift-service-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:17.007447 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:17.007416 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb707642-c29d-4fee-9070-06e12a582b55-metrics-certs" pod "fb707642-c29d-4fee-9070-06e12a582b55" (UID: "fb707642-c29d-4fee-9070-06e12a582b55"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:17.007556 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:17.007461 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb707642-c29d-4fee-9070-06e12a582b55-maas-api-tls" pod "fb707642-c29d-4fee-9070-06e12a582b55" (UID: "fb707642-c29d-4fee-9070-06e12a582b55"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:17.007556 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:17.007524 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb707642-c29d-4fee-9070-06e12a582b55-kube-api-access-vwmf2" pod "fb707642-c29d-4fee-9070-06e12a582b55" (UID: "fb707642-c29d-4fee-9070-06e12a582b55"). InnerVolumeSpecName "kube-api-access-vwmf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:17.108293 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:17.108258 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwmf2\" (UniqueName: \"kubernetes.io/projected/fb707642-c29d-4fee-9070-06e12a582b55-kube-api-access-vwmf2\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:17.108293 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:17.108291 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb707642-c29d-4fee-9070-06e12a582b55-maas-api-tls\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:17.108293 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:17.108301 7059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb707642-c29d-4fee-9070-06e12a582b55-metrics-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:17.892474 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:17.892442 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-385061-55d99c47d4-6mktg" Nov 04 01:33:17.892999 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:17.892496 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-385061-55d99c47d4-6mktg" event={"ID":"fb707642-c29d-4fee-9070-06e12a582b55","Type":"ContainerDied","Data":"e624a2773c77251f2d03f09b804ca136c9ff15eef8cd92490bf466731964df39"} Nov 04 01:33:17.892999 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:17.892559 7059 scope.go:122] "RemoveContainer" containerID="0b0b2d0cdf5fafa87a505fe7ec4dca18eff0d75c1eaeb0622a29f8b96deef8d2" Nov 04 01:33:17.919550 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:17.919520 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-385061-55d99c47d4-6mktg"] Nov 04 01:33:17.928268 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:17.928245 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-385061-55d99c47d4-6mktg"] Nov 04 01:33:19.180385 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:19.180346 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb707642-c29d-4fee-9070-06e12a582b55" path="/var/lib/kubelet/pods/fb707642-c29d-4fee-9070-06e12a582b55/volumes" Nov 04 01:33:21.353970 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:33:21.353886 7059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod612b8bca_a31e_4126_a517_ff449f55acd8.slice/crio-conmon-ba4e37ce5194dddbbc75cbaeecb38504bd9e8b011e116861608b683a10b3604b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod612b8bca_a31e_4126_a517_ff449f55acd8.slice/crio-ba4e37ce5194dddbbc75cbaeecb38504bd9e8b011e116861608b683a10b3604b.scope\": RecentStats: unable to find data in memory cache]" Nov 04 01:33:21.567247 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.567199 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:33:21.658388 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.658347 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-workload-socket\") pod \"b8b355cb-9a0e-45fe-8928-850062b32cdb\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " Nov 04 01:33:21.658586 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.658407 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-istio-data\") pod \"b8b355cb-9a0e-45fe-8928-850062b32cdb\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " Nov 04 01:33:21.658586 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.658456 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8b355cb-9a0e-45fe-8928-850062b32cdb-istio-token\" (UniqueName: \"kubernetes.io/projected/b8b355cb-9a0e-45fe-8928-850062b32cdb-istio-token\") pod \"b8b355cb-9a0e-45fe-8928-850062b32cdb\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " Nov 04 01:33:21.658586 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.658524 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8b355cb-9a0e-45fe-8928-850062b32cdb-kube-api-access-47xmf\" (UniqueName: \"kubernetes.io/projected/b8b355cb-9a0e-45fe-8928-850062b32cdb-kube-api-access-47xmf\") pod \"b8b355cb-9a0e-45fe-8928-850062b32cdb\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " Nov 04 01:33:21.658778 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.658632 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8b355cb-9a0e-45fe-8928-850062b32cdb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b355cb-9a0e-45fe-8928-850062b32cdb-istio-podinfo\") pod \"b8b355cb-9a0e-45fe-8928-850062b32cdb\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " Nov 04 01:33:21.658778 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.658746 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-workload-certs\") pod \"b8b355cb-9a0e-45fe-8928-850062b32cdb\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " Nov 04 01:33:21.658886 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.658809 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-istio-envoy\") pod \"b8b355cb-9a0e-45fe-8928-850062b32cdb\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " Nov 04 01:33:21.658886 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.658836 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-credential-socket\") pod \"b8b355cb-9a0e-45fe-8928-850062b32cdb\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " Nov 04 01:33:21.659010 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.658914 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8b355cb-9a0e-45fe-8928-850062b32cdb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b355cb-9a0e-45fe-8928-850062b32cdb-istiod-ca-cert\") pod \"b8b355cb-9a0e-45fe-8928-850062b32cdb\" (UID: \"b8b355cb-9a0e-45fe-8928-850062b32cdb\") " Nov 04 01:33:21.660804 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.660688 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-credential-socket" pod "b8b355cb-9a0e-45fe-8928-850062b32cdb" (UID: "b8b355cb-9a0e-45fe-8928-850062b32cdb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:21.660804 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.660758 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-workload-socket" pod "b8b355cb-9a0e-45fe-8928-850062b32cdb" (UID: "b8b355cb-9a0e-45fe-8928-850062b32cdb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:21.661023 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.660821 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-workload-certs" pod "b8b355cb-9a0e-45fe-8928-850062b32cdb" (UID: "b8b355cb-9a0e-45fe-8928-850062b32cdb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:21.661121 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.661038 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-istio-data" pod "b8b355cb-9a0e-45fe-8928-850062b32cdb" (UID: "b8b355cb-9a0e-45fe-8928-850062b32cdb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:21.664603 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.661227 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b355cb-9a0e-45fe-8928-850062b32cdb-istiod-ca-cert" pod "b8b355cb-9a0e-45fe-8928-850062b32cdb" (UID: "b8b355cb-9a0e-45fe-8928-850062b32cdb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:21.664603 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.664180 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8b355cb-9a0e-45fe-8928-850062b32cdb-istio-podinfo" pod "b8b355cb-9a0e-45fe-8928-850062b32cdb" (UID: "b8b355cb-9a0e-45fe-8928-850062b32cdb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:21.670264 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.670229 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b355cb-9a0e-45fe-8928-850062b32cdb-istio-token" pod "b8b355cb-9a0e-45fe-8928-850062b32cdb" (UID: "b8b355cb-9a0e-45fe-8928-850062b32cdb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:21.672113 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.672085 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b355cb-9a0e-45fe-8928-850062b32cdb-kube-api-access-47xmf" pod "b8b355cb-9a0e-45fe-8928-850062b32cdb" (UID: "b8b355cb-9a0e-45fe-8928-850062b32cdb"). InnerVolumeSpecName "kube-api-access-47xmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:21.672275 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.672252 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-istio-envoy" pod "b8b355cb-9a0e-45fe-8928-850062b32cdb" (UID: "b8b355cb-9a0e-45fe-8928-850062b32cdb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:21.724306 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.724279 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:33:21.761387 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.761342 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/612b8bca-a31e-4126-a517-ff449f55acd8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/612b8bca-a31e-4126-a517-ff449f55acd8-istio-podinfo\") pod \"612b8bca-a31e-4126-a517-ff449f55acd8\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " Nov 04 01:33:21.761552 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.761493 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-istio-envoy\") pod \"612b8bca-a31e-4126-a517-ff449f55acd8\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " Nov 04 01:33:21.761623 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.761555 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/612b8bca-a31e-4126-a517-ff449f55acd8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/612b8bca-a31e-4126-a517-ff449f55acd8-istiod-ca-cert\") pod \"612b8bca-a31e-4126-a517-ff449f55acd8\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " Nov 04 01:33:21.761623 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.761591 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-workload-certs\") pod \"612b8bca-a31e-4126-a517-ff449f55acd8\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " Nov 04 01:33:21.761727 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.761656 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-istio-data\") pod \"612b8bca-a31e-4126-a517-ff449f55acd8\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " Nov 04 01:33:21.761793 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.761723 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/612b8bca-a31e-4126-a517-ff449f55acd8-istio-token\" (UniqueName: \"kubernetes.io/projected/612b8bca-a31e-4126-a517-ff449f55acd8-istio-token\") pod \"612b8bca-a31e-4126-a517-ff449f55acd8\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " Nov 04 01:33:21.761848 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.761830 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/612b8bca-a31e-4126-a517-ff449f55acd8-kube-api-access-tx2tr\" (UniqueName: \"kubernetes.io/projected/612b8bca-a31e-4126-a517-ff449f55acd8-kube-api-access-tx2tr\") pod \"612b8bca-a31e-4126-a517-ff449f55acd8\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " Nov 04 01:33:21.761903 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.761888 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-workload-socket\") pod \"612b8bca-a31e-4126-a517-ff449f55acd8\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " Nov 04 01:33:21.762021 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.761983 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-credential-socket\") pod \"612b8bca-a31e-4126-a517-ff449f55acd8\" (UID: \"612b8bca-a31e-4126-a517-ff449f55acd8\") " Nov 04 01:33:21.762125 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.762088 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/612b8bca-a31e-4126-a517-ff449f55acd8-istiod-ca-cert" pod "612b8bca-a31e-4126-a517-ff449f55acd8" (UID: "612b8bca-a31e-4126-a517-ff449f55acd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:21.762852 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.762824 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b355cb-9a0e-45fe-8928-850062b32cdb-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:21.762980 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.762856 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/612b8bca-a31e-4126-a517-ff449f55acd8-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:21.762980 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.762869 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:21.762980 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.762885 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:21.762980 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.762899 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:21.762980 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.762914 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b355cb-9a0e-45fe-8928-850062b32cdb-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:21.762980 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.762947 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:21.762980 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.762962 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b355cb-9a0e-45fe-8928-850062b32cdb-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:21.762980 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.762976 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8b355cb-9a0e-45fe-8928-850062b32cdb-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:21.763409 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.762990 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47xmf\" (UniqueName: \"kubernetes.io/projected/b8b355cb-9a0e-45fe-8928-850062b32cdb-kube-api-access-47xmf\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:21.763409 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.763138 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-istio-data" pod "612b8bca-a31e-4126-a517-ff449f55acd8" (UID: "612b8bca-a31e-4126-a517-ff449f55acd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:21.763409 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.763250 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-credential-socket" pod "612b8bca-a31e-4126-a517-ff449f55acd8" (UID: "612b8bca-a31e-4126-a517-ff449f55acd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:21.763572 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.763485 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-workload-certs" pod "612b8bca-a31e-4126-a517-ff449f55acd8" (UID: "612b8bca-a31e-4126-a517-ff449f55acd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:21.763572 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.763501 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-workload-socket" pod "612b8bca-a31e-4126-a517-ff449f55acd8" (UID: "612b8bca-a31e-4126-a517-ff449f55acd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:21.764201 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.764172 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/612b8bca-a31e-4126-a517-ff449f55acd8-istio-podinfo" pod "612b8bca-a31e-4126-a517-ff449f55acd8" (UID: "612b8bca-a31e-4126-a517-ff449f55acd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:21.765496 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.765471 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/612b8bca-a31e-4126-a517-ff449f55acd8-istio-token" pod "612b8bca-a31e-4126-a517-ff449f55acd8" (UID: "612b8bca-a31e-4126-a517-ff449f55acd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:21.765614 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.765537 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-istio-envoy" pod "612b8bca-a31e-4126-a517-ff449f55acd8" (UID: "612b8bca-a31e-4126-a517-ff449f55acd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:21.767486 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.767465 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/612b8bca-a31e-4126-a517-ff449f55acd8-kube-api-access-tx2tr" pod "612b8bca-a31e-4126-a517-ff449f55acd8" (UID: "612b8bca-a31e-4126-a517-ff449f55acd8"). InnerVolumeSpecName "kube-api-access-tx2tr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:21.864821 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.864733 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/612b8bca-a31e-4126-a517-ff449f55acd8-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:21.864821 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.864764 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx2tr\" (UniqueName: \"kubernetes.io/projected/612b8bca-a31e-4126-a517-ff449f55acd8-kube-api-access-tx2tr\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:21.864821 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.864777 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:21.864821 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.864791 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:21.864821 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.864803 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/612b8bca-a31e-4126-a517-ff449f55acd8-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:21.864821 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.864816 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:21.865158 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.864830 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:21.865158 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:21.864839 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/612b8bca-a31e-4126-a517-ff449f55acd8-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.036546 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:22.036500 7059 generic.go:356] "Generic (PLEG): container finished" podID="b8b355cb-9a0e-45fe-8928-850062b32cdb" containerID="e0fd804baf641c6572e9584fffc24401af9586b278d832a34551b65ef8ce7910" exitCode=0 Nov 04 01:33:22.036740 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:22.036679 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" event={"ID":"b8b355cb-9a0e-45fe-8928-850062b32cdb","Type":"ContainerDied","Data":"e0fd804baf641c6572e9584fffc24401af9586b278d832a34551b65ef8ce7910"} Nov 04 01:33:22.036821 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:22.036779 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" event={"ID":"b8b355cb-9a0e-45fe-8928-850062b32cdb","Type":"ContainerDied","Data":"ca8e04ce49f60773b07fa32ed07f183c31c8dc478162868b2e1483d1e830dacd"} Nov 04 01:33:22.036880 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:22.036826 7059 scope.go:122] "RemoveContainer" containerID="e0fd804baf641c6572e9584fffc24401af9586b278d832a34551b65ef8ce7910" Nov 04 01:33:22.046046 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:22.037064 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k" Nov 04 01:33:22.053638 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:22.053603 7059 generic.go:356] "Generic (PLEG): container finished" podID="612b8bca-a31e-4126-a517-ff449f55acd8" containerID="ba4e37ce5194dddbbc75cbaeecb38504bd9e8b011e116861608b683a10b3604b" exitCode=0 Nov 04 01:33:22.053760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:22.053675 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" Nov 04 01:33:22.053827 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:22.053681 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" event={"ID":"612b8bca-a31e-4126-a517-ff449f55acd8","Type":"ContainerDied","Data":"ba4e37ce5194dddbbc75cbaeecb38504bd9e8b011e116861608b683a10b3604b"} Nov 04 01:33:22.053827 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:22.053796 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb" event={"ID":"612b8bca-a31e-4126-a517-ff449f55acd8","Type":"ContainerDied","Data":"48976f41dccc87c032e353d388819ea71d843994ea19e60183dcdc520c587549"} Nov 04 01:33:22.065538 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:22.065512 7059 scope.go:122] "RemoveContainer" containerID="e0fd804baf641c6572e9584fffc24401af9586b278d832a34551b65ef8ce7910" Nov 04 01:33:22.065889 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:33:22.065866 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0fd804baf641c6572e9584fffc24401af9586b278d832a34551b65ef8ce7910\": container with ID starting with e0fd804baf641c6572e9584fffc24401af9586b278d832a34551b65ef8ce7910 not found: ID does not exist" containerID="e0fd804baf641c6572e9584fffc24401af9586b278d832a34551b65ef8ce7910" Nov 04 01:33:22.066017 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:22.065902 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0fd804baf641c6572e9584fffc24401af9586b278d832a34551b65ef8ce7910"} err="failed to get container status \"e0fd804baf641c6572e9584fffc24401af9586b278d832a34551b65ef8ce7910\": rpc error: code = NotFound desc = could not find container \"e0fd804baf641c6572e9584fffc24401af9586b278d832a34551b65ef8ce7910\": container with ID starting with e0fd804baf641c6572e9584fffc24401af9586b278d832a34551b65ef8ce7910 not found: ID does not exist" Nov 04 01:33:22.066017 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:22.065943 7059 scope.go:122] "RemoveContainer" containerID="ba4e37ce5194dddbbc75cbaeecb38504bd9e8b011e116861608b683a10b3604b" Nov 04 01:33:22.078017 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:22.077994 7059 scope.go:122] "RemoveContainer" containerID="ba4e37ce5194dddbbc75cbaeecb38504bd9e8b011e116861608b683a10b3604b" Nov 04 01:33:22.078300 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:33:22.078279 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba4e37ce5194dddbbc75cbaeecb38504bd9e8b011e116861608b683a10b3604b\": container with ID starting with ba4e37ce5194dddbbc75cbaeecb38504bd9e8b011e116861608b683a10b3604b not found: ID does not exist" containerID="ba4e37ce5194dddbbc75cbaeecb38504bd9e8b011e116861608b683a10b3604b" Nov 04 01:33:22.078382 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:22.078306 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba4e37ce5194dddbbc75cbaeecb38504bd9e8b011e116861608b683a10b3604b"} err="failed to get container status \"ba4e37ce5194dddbbc75cbaeecb38504bd9e8b011e116861608b683a10b3604b\": rpc error: code = NotFound desc = could not find container \"ba4e37ce5194dddbbc75cbaeecb38504bd9e8b011e116861608b683a10b3604b\": container with ID starting with ba4e37ce5194dddbbc75cbaeecb38504bd9e8b011e116861608b683a10b3604b not found: ID does not exist" Nov 04 01:33:22.098901 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:22.098873 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k"] Nov 04 01:33:22.113890 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:22.113863 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-wm24k"] Nov 04 01:33:22.145474 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:22.145404 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb"] Nov 04 01:33:22.154724 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:22.154693 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-bd8jb"] Nov 04 01:33:23.186245 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:23.186214 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="612b8bca-a31e-4126-a517-ff449f55acd8" path="/var/lib/kubelet/pods/612b8bca-a31e-4126-a517-ff449f55acd8/volumes" Nov 04 01:33:23.186809 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:23.186793 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8b355cb-9a0e-45fe-8928-850062b32cdb" path="/var/lib/kubelet/pods/b8b355cb-9a0e-45fe-8928-850062b32cdb/volumes" Nov 04 01:33:23.549864 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:23.549779 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl"] Nov 04 01:33:23.550515 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:23.550036 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl" podUID="bfeb8a23-780b-4ac0-9308-afdd49e42175" containerName="maas-api" containerID="cri-o://47f07174dd08d06d4e525eb0ae4e28bec64e3a0c0c8c6241be8db278d679b1c6" gracePeriod=30 Nov 04 01:33:24.065701 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:24.065658 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl" Nov 04 01:33:24.104360 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:24.104322 7059 generic.go:356] "Generic (PLEG): container finished" podID="bfeb8a23-780b-4ac0-9308-afdd49e42175" containerID="47f07174dd08d06d4e525eb0ae4e28bec64e3a0c0c8c6241be8db278d679b1c6" exitCode=0 Nov 04 01:33:24.104531 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:24.104433 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl" event={"ID":"bfeb8a23-780b-4ac0-9308-afdd49e42175","Type":"ContainerDied","Data":"47f07174dd08d06d4e525eb0ae4e28bec64e3a0c0c8c6241be8db278d679b1c6"} Nov 04 01:33:24.104531 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:24.104484 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl" event={"ID":"bfeb8a23-780b-4ac0-9308-afdd49e42175","Type":"ContainerDied","Data":"f101e161ca84aeba61566ed175911775ab17f110f565f8110a4c211137aaaedb"} Nov 04 01:33:24.104531 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:24.104509 7059 scope.go:122] "RemoveContainer" containerID="47f07174dd08d06d4e525eb0ae4e28bec64e3a0c0c8c6241be8db278d679b1c6" Nov 04 01:33:24.104676 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:24.104607 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl" Nov 04 01:33:24.121047 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:24.121022 7059 scope.go:122] "RemoveContainer" containerID="47f07174dd08d06d4e525eb0ae4e28bec64e3a0c0c8c6241be8db278d679b1c6" Nov 04 01:33:24.121384 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:33:24.121357 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47f07174dd08d06d4e525eb0ae4e28bec64e3a0c0c8c6241be8db278d679b1c6\": container with ID starting with 47f07174dd08d06d4e525eb0ae4e28bec64e3a0c0c8c6241be8db278d679b1c6 not found: ID does not exist" containerID="47f07174dd08d06d4e525eb0ae4e28bec64e3a0c0c8c6241be8db278d679b1c6" Nov 04 01:33:24.121499 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:24.121399 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f07174dd08d06d4e525eb0ae4e28bec64e3a0c0c8c6241be8db278d679b1c6"} err="failed to get container status \"47f07174dd08d06d4e525eb0ae4e28bec64e3a0c0c8c6241be8db278d679b1c6\": rpc error: code = NotFound desc = could not find container \"47f07174dd08d06d4e525eb0ae4e28bec64e3a0c0c8c6241be8db278d679b1c6\": container with ID starting with 47f07174dd08d06d4e525eb0ae4e28bec64e3a0c0c8c6241be8db278d679b1c6 not found: ID does not exist" Nov 04 01:33:24.140990 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:24.140963 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfeb8a23-780b-4ac0-9308-afdd49e42175-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfeb8a23-780b-4ac0-9308-afdd49e42175-maas-api-tls\") pod \"bfeb8a23-780b-4ac0-9308-afdd49e42175\" (UID: \"bfeb8a23-780b-4ac0-9308-afdd49e42175\") " Nov 04 01:33:24.141166 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:24.141149 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfeb8a23-780b-4ac0-9308-afdd49e42175-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfeb8a23-780b-4ac0-9308-afdd49e42175-metrics-certs\") pod \"bfeb8a23-780b-4ac0-9308-afdd49e42175\" (UID: \"bfeb8a23-780b-4ac0-9308-afdd49e42175\") " Nov 04 01:33:24.141250 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:24.141197 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfeb8a23-780b-4ac0-9308-afdd49e42175-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfeb8a23-780b-4ac0-9308-afdd49e42175-openshift-service-ca\") pod \"bfeb8a23-780b-4ac0-9308-afdd49e42175\" (UID: \"bfeb8a23-780b-4ac0-9308-afdd49e42175\") " Nov 04 01:33:24.141419 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:24.141399 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfeb8a23-780b-4ac0-9308-afdd49e42175-kube-api-access-mr6c6\" (UniqueName: \"kubernetes.io/projected/bfeb8a23-780b-4ac0-9308-afdd49e42175-kube-api-access-mr6c6\") pod \"bfeb8a23-780b-4ac0-9308-afdd49e42175\" (UID: \"bfeb8a23-780b-4ac0-9308-afdd49e42175\") " Nov 04 01:33:24.141581 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:24.141554 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfeb8a23-780b-4ac0-9308-afdd49e42175-openshift-service-ca" pod "bfeb8a23-780b-4ac0-9308-afdd49e42175" (UID: "bfeb8a23-780b-4ac0-9308-afdd49e42175"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:24.142419 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:24.142391 7059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfeb8a23-780b-4ac0-9308-afdd49e42175-openshift-service-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:24.144715 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:24.144692 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfeb8a23-780b-4ac0-9308-afdd49e42175-kube-api-access-mr6c6" pod "bfeb8a23-780b-4ac0-9308-afdd49e42175" (UID: "bfeb8a23-780b-4ac0-9308-afdd49e42175"). InnerVolumeSpecName "kube-api-access-mr6c6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:24.145166 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:24.145137 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfeb8a23-780b-4ac0-9308-afdd49e42175-maas-api-tls" pod "bfeb8a23-780b-4ac0-9308-afdd49e42175" (UID: "bfeb8a23-780b-4ac0-9308-afdd49e42175"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:24.146568 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:24.146547 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfeb8a23-780b-4ac0-9308-afdd49e42175-metrics-certs" pod "bfeb8a23-780b-4ac0-9308-afdd49e42175" (UID: "bfeb8a23-780b-4ac0-9308-afdd49e42175"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:24.244648 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:24.244609 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr6c6\" (UniqueName: \"kubernetes.io/projected/bfeb8a23-780b-4ac0-9308-afdd49e42175-kube-api-access-mr6c6\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:24.244648 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:24.244647 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfeb8a23-780b-4ac0-9308-afdd49e42175-maas-api-tls\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:24.245148 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:24.244661 7059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfeb8a23-780b-4ac0-9308-afdd49e42175-metrics-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:24.469529 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:24.469489 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl"] Nov 04 01:33:24.479369 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:24.479341 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4f8dd0-665f589f57-rmttl"] Nov 04 01:33:25.180432 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:25.180398 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfeb8a23-780b-4ac0-9308-afdd49e42175" path="/var/lib/kubelet/pods/bfeb8a23-780b-4ac0-9308-afdd49e42175/volumes" Nov 04 01:33:29.137331 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:29.136160 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9"] Nov 04 01:33:29.137331 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:29.136407 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" podUID="7f3f33a9-ead2-4fb8-b066-5a45f35881c9" containerName="istio-proxy" containerID="cri-o://4c75c8794daef6c19284f9044940dc0b1f6d1566219ac71bb0a768ec8d11031b" gracePeriod=30 Nov 04 01:33:29.535949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:29.535522 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876"] Nov 04 01:33:29.536159 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:29.535967 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" podUID="49cff42c-6b14-409e-b5a1-9eb70c788da9" containerName="istio-proxy" containerID="cri-o://bb8bfcd43e9b13d72b47ab8700ac14fc4d34a7712593d4f2cb2666457f8d177b" gracePeriod=30 Nov 04 01:33:32.190990 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:32.190957 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt"] Nov 04 01:33:32.191480 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:32.191191 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt" podUID="cd37362b-7164-4401-bd4b-08af565a8bc4" containerName="maas-api" containerID="cri-o://725f457b7eb25f2c353d7a183873b78ce64565ffb92bcb58afacd4564d163a65" gracePeriod=30 Nov 04 01:33:32.363046 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:32.363001 7059 generic.go:356] "Generic (PLEG): container finished" podID="cd37362b-7164-4401-bd4b-08af565a8bc4" containerID="725f457b7eb25f2c353d7a183873b78ce64565ffb92bcb58afacd4564d163a65" exitCode=0 Nov 04 01:33:32.363274 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:32.363104 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt" event={"ID":"cd37362b-7164-4401-bd4b-08af565a8bc4","Type":"ContainerDied","Data":"725f457b7eb25f2c353d7a183873b78ce64565ffb92bcb58afacd4564d163a65"} Nov 04 01:33:32.684951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:32.684117 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt" Nov 04 01:33:32.748863 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:32.748709 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd37362b-7164-4401-bd4b-08af565a8bc4-kube-api-access-w5n4r\" (UniqueName: \"kubernetes.io/projected/cd37362b-7164-4401-bd4b-08af565a8bc4-kube-api-access-w5n4r\") pod \"cd37362b-7164-4401-bd4b-08af565a8bc4\" (UID: \"cd37362b-7164-4401-bd4b-08af565a8bc4\") " Nov 04 01:33:32.749218 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:32.749196 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd37362b-7164-4401-bd4b-08af565a8bc4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd37362b-7164-4401-bd4b-08af565a8bc4-openshift-service-ca\") pod \"cd37362b-7164-4401-bd4b-08af565a8bc4\" (UID: \"cd37362b-7164-4401-bd4b-08af565a8bc4\") " Nov 04 01:33:32.749370 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:32.749353 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd37362b-7164-4401-bd4b-08af565a8bc4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd37362b-7164-4401-bd4b-08af565a8bc4-maas-api-tls\") pod \"cd37362b-7164-4401-bd4b-08af565a8bc4\" (UID: \"cd37362b-7164-4401-bd4b-08af565a8bc4\") " Nov 04 01:33:32.749486 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:32.749473 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd37362b-7164-4401-bd4b-08af565a8bc4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd37362b-7164-4401-bd4b-08af565a8bc4-metrics-certs\") pod \"cd37362b-7164-4401-bd4b-08af565a8bc4\" (UID: \"cd37362b-7164-4401-bd4b-08af565a8bc4\") " Nov 04 01:33:32.750311 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:32.750276 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd37362b-7164-4401-bd4b-08af565a8bc4-openshift-service-ca" pod "cd37362b-7164-4401-bd4b-08af565a8bc4" (UID: "cd37362b-7164-4401-bd4b-08af565a8bc4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:32.772753 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:32.772692 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd37362b-7164-4401-bd4b-08af565a8bc4-maas-api-tls" pod "cd37362b-7164-4401-bd4b-08af565a8bc4" (UID: "cd37362b-7164-4401-bd4b-08af565a8bc4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:32.785401 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:32.783803 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd37362b-7164-4401-bd4b-08af565a8bc4-kube-api-access-w5n4r" pod "cd37362b-7164-4401-bd4b-08af565a8bc4" (UID: "cd37362b-7164-4401-bd4b-08af565a8bc4"). InnerVolumeSpecName "kube-api-access-w5n4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:32.787529 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:32.787498 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd37362b-7164-4401-bd4b-08af565a8bc4-metrics-certs" pod "cd37362b-7164-4401-bd4b-08af565a8bc4" (UID: "cd37362b-7164-4401-bd4b-08af565a8bc4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:32.852199 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:32.852163 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5n4r\" (UniqueName: \"kubernetes.io/projected/cd37362b-7164-4401-bd4b-08af565a8bc4-kube-api-access-w5n4r\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:32.852683 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:32.852524 7059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd37362b-7164-4401-bd4b-08af565a8bc4-openshift-service-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:32.853033 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:32.853015 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd37362b-7164-4401-bd4b-08af565a8bc4-maas-api-tls\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:32.853150 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:32.853139 7059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd37362b-7164-4401-bd4b-08af565a8bc4-metrics-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.375201 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:33.375171 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt" Nov 04 01:33:33.375697 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:33.375172 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt" event={"ID":"cd37362b-7164-4401-bd4b-08af565a8bc4","Type":"ContainerDied","Data":"bc56e19d05e9d00125aafe85c3cccc71cc4cbae405048255780325aeac8847ae"} Nov 04 01:33:33.375697 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:33.375307 7059 scope.go:122] "RemoveContainer" containerID="725f457b7eb25f2c353d7a183873b78ce64565ffb92bcb58afacd4564d163a65" Nov 04 01:33:33.413853 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:33.413781 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt"] Nov 04 01:33:33.417793 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:33.417763 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5f572a-5d96d9dd5c-c69rt"] Nov 04 01:33:34.427997 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:34.424902 7059 generic.go:356] "Generic (PLEG): container finished" podID="7f3f33a9-ead2-4fb8-b066-5a45f35881c9" containerID="4c75c8794daef6c19284f9044940dc0b1f6d1566219ac71bb0a768ec8d11031b" exitCode=0 Nov 04 01:33:34.427997 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:34.425139 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" event={"ID":"7f3f33a9-ead2-4fb8-b066-5a45f35881c9","Type":"ContainerDied","Data":"4c75c8794daef6c19284f9044940dc0b1f6d1566219ac71bb0a768ec8d11031b"} Nov 04 01:33:34.572195 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:34.572166 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:33:34.713506 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:34.713469 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-credential-socket\") pod \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " Nov 04 01:33:34.713810 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:34.713622 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istio-token\" (UniqueName: \"kubernetes.io/projected/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istio-token\") pod \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " Nov 04 01:33:34.713810 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:34.713675 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-kube-api-access-l74qr\" (UniqueName: \"kubernetes.io/projected/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-kube-api-access-l74qr\") pod \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " Nov 04 01:33:34.713810 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:34.713744 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istio-data\") pod \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " Nov 04 01:33:34.714059 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:34.713813 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-workload-certs\") pod \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " Nov 04 01:33:34.714059 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:34.713865 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istio-envoy\") pod \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " Nov 04 01:33:34.714059 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:34.713903 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-workload-socket\") pod \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " Nov 04 01:33:34.714059 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:34.713940 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istiod-ca-cert\") pod \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " Nov 04 01:33:34.714059 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:34.713969 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istio-podinfo\") pod \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\" (UID: \"7f3f33a9-ead2-4fb8-b066-5a45f35881c9\") " Nov 04 01:33:34.715721 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:34.715036 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istio-data" pod "7f3f33a9-ead2-4fb8-b066-5a45f35881c9" (UID: "7f3f33a9-ead2-4fb8-b066-5a45f35881c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:34.715721 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:34.715057 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-workload-certs" pod "7f3f33a9-ead2-4fb8-b066-5a45f35881c9" (UID: "7f3f33a9-ead2-4fb8-b066-5a45f35881c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:34.715721 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:34.715290 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-credential-socket" pod "7f3f33a9-ead2-4fb8-b066-5a45f35881c9" (UID: "7f3f33a9-ead2-4fb8-b066-5a45f35881c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:34.715721 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:34.715519 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-workload-socket" pod "7f3f33a9-ead2-4fb8-b066-5a45f35881c9" (UID: "7f3f33a9-ead2-4fb8-b066-5a45f35881c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:34.715721 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:34.715606 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istiod-ca-cert" pod "7f3f33a9-ead2-4fb8-b066-5a45f35881c9" (UID: "7f3f33a9-ead2-4fb8-b066-5a45f35881c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:34.717099 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:34.716998 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istio-podinfo" pod "7f3f33a9-ead2-4fb8-b066-5a45f35881c9" (UID: "7f3f33a9-ead2-4fb8-b066-5a45f35881c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:34.718078 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:34.718045 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-kube-api-access-l74qr" pod "7f3f33a9-ead2-4fb8-b066-5a45f35881c9" (UID: "7f3f33a9-ead2-4fb8-b066-5a45f35881c9"). InnerVolumeSpecName "kube-api-access-l74qr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:34.719113 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:34.719090 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istio-envoy" pod "7f3f33a9-ead2-4fb8-b066-5a45f35881c9" (UID: "7f3f33a9-ead2-4fb8-b066-5a45f35881c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:34.719284 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:34.719261 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istio-token" pod "7f3f33a9-ead2-4fb8-b066-5a45f35881c9" (UID: "7f3f33a9-ead2-4fb8-b066-5a45f35881c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:34.865996 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:34.865351 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:34.865996 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:34.865390 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:34.865996 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:34.865403 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:34.865996 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:34.865415 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l74qr\" (UniqueName: \"kubernetes.io/projected/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-kube-api-access-l74qr\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:34.865996 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:34.865433 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:34.865996 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:34.865445 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:34.865996 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:34.865458 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:34.865996 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:34.865471 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:34.865996 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:34.865483 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f3f33a9-ead2-4fb8-b066-5a45f35881c9-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.068020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.066857 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:33:35.179180 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.179153 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-workload-certs\") pod \"49cff42c-6b14-409e-b5a1-9eb70c788da9\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " Nov 04 01:33:35.179510 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.179201 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-workload-socket\") pod \"49cff42c-6b14-409e-b5a1-9eb70c788da9\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " Nov 04 01:33:35.179510 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.179225 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49cff42c-6b14-409e-b5a1-9eb70c788da9-istio-token\" (UniqueName: \"kubernetes.io/projected/49cff42c-6b14-409e-b5a1-9eb70c788da9-istio-token\") pod \"49cff42c-6b14-409e-b5a1-9eb70c788da9\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " Nov 04 01:33:35.179510 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.179308 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49cff42c-6b14-409e-b5a1-9eb70c788da9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49cff42c-6b14-409e-b5a1-9eb70c788da9-istio-podinfo\") pod \"49cff42c-6b14-409e-b5a1-9eb70c788da9\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " Nov 04 01:33:35.179510 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.179395 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-istio-envoy\") pod \"49cff42c-6b14-409e-b5a1-9eb70c788da9\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " Nov 04 01:33:35.179510 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.179438 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49cff42c-6b14-409e-b5a1-9eb70c788da9-kube-api-access-grfhn\" (UniqueName: \"kubernetes.io/projected/49cff42c-6b14-409e-b5a1-9eb70c788da9-kube-api-access-grfhn\") pod \"49cff42c-6b14-409e-b5a1-9eb70c788da9\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " Nov 04 01:33:35.179510 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.179502 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-workload-certs" pod "49cff42c-6b14-409e-b5a1-9eb70c788da9" (UID: "49cff42c-6b14-409e-b5a1-9eb70c788da9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:35.179891 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.179559 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49cff42c-6b14-409e-b5a1-9eb70c788da9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49cff42c-6b14-409e-b5a1-9eb70c788da9-istiod-ca-cert\") pod \"49cff42c-6b14-409e-b5a1-9eb70c788da9\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " Nov 04 01:33:35.179891 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.179674 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-istio-data\") pod \"49cff42c-6b14-409e-b5a1-9eb70c788da9\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " Nov 04 01:33:35.179891 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.179818 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-credential-socket\") pod \"49cff42c-6b14-409e-b5a1-9eb70c788da9\" (UID: \"49cff42c-6b14-409e-b5a1-9eb70c788da9\") " Nov 04 01:33:35.180104 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.180076 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49cff42c-6b14-409e-b5a1-9eb70c788da9-istiod-ca-cert" pod "49cff42c-6b14-409e-b5a1-9eb70c788da9" (UID: "49cff42c-6b14-409e-b5a1-9eb70c788da9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:35.180390 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.180334 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-istio-data" pod "49cff42c-6b14-409e-b5a1-9eb70c788da9" (UID: "49cff42c-6b14-409e-b5a1-9eb70c788da9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:35.180809 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.180783 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-credential-socket" pod "49cff42c-6b14-409e-b5a1-9eb70c788da9" (UID: "49cff42c-6b14-409e-b5a1-9eb70c788da9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:35.181116 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.181040 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-workload-socket" pod "49cff42c-6b14-409e-b5a1-9eb70c788da9" (UID: "49cff42c-6b14-409e-b5a1-9eb70c788da9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:35.181254 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.181234 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49cff42c-6b14-409e-b5a1-9eb70c788da9-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.181792 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.181768 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.181904 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.181826 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.181904 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.181842 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.181904 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.181859 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.182088 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.182037 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd37362b-7164-4401-bd4b-08af565a8bc4" path="/var/lib/kubelet/pods/cd37362b-7164-4401-bd4b-08af565a8bc4/volumes" Nov 04 01:33:35.182447 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.182428 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49cff42c-6b14-409e-b5a1-9eb70c788da9-istio-token" pod "49cff42c-6b14-409e-b5a1-9eb70c788da9" (UID: "49cff42c-6b14-409e-b5a1-9eb70c788da9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:35.183273 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.183249 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-istio-envoy" pod "49cff42c-6b14-409e-b5a1-9eb70c788da9" (UID: "49cff42c-6b14-409e-b5a1-9eb70c788da9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:35.183359 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.183269 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49cff42c-6b14-409e-b5a1-9eb70c788da9-istio-podinfo" pod "49cff42c-6b14-409e-b5a1-9eb70c788da9" (UID: "49cff42c-6b14-409e-b5a1-9eb70c788da9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:35.184681 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.184654 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49cff42c-6b14-409e-b5a1-9eb70c788da9-kube-api-access-grfhn" pod "49cff42c-6b14-409e-b5a1-9eb70c788da9" (UID: "49cff42c-6b14-409e-b5a1-9eb70c788da9"). InnerVolumeSpecName "kube-api-access-grfhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:35.283048 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.283022 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49cff42c-6b14-409e-b5a1-9eb70c788da9-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.283048 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.283048 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49cff42c-6b14-409e-b5a1-9eb70c788da9-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.283221 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.283058 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49cff42c-6b14-409e-b5a1-9eb70c788da9-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.283221 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.283067 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grfhn\" (UniqueName: \"kubernetes.io/projected/49cff42c-6b14-409e-b5a1-9eb70c788da9-kube-api-access-grfhn\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.437591 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.437517 7059 generic.go:356] "Generic (PLEG): container finished" podID="49cff42c-6b14-409e-b5a1-9eb70c788da9" containerID="bb8bfcd43e9b13d72b47ab8700ac14fc4d34a7712593d4f2cb2666457f8d177b" exitCode=0 Nov 04 01:33:35.437591 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.437567 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" Nov 04 01:33:35.438076 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.437590 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" event={"ID":"49cff42c-6b14-409e-b5a1-9eb70c788da9","Type":"ContainerDied","Data":"bb8bfcd43e9b13d72b47ab8700ac14fc4d34a7712593d4f2cb2666457f8d177b"} Nov 04 01:33:35.438076 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.437646 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876" event={"ID":"49cff42c-6b14-409e-b5a1-9eb70c788da9","Type":"ContainerDied","Data":"d667ab40fd561489cd4d4520835597e4138d466ef97bd51a2d5ff1cc9a31af24"} Nov 04 01:33:35.438076 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.437670 7059 scope.go:122] "RemoveContainer" containerID="bb8bfcd43e9b13d72b47ab8700ac14fc4d34a7712593d4f2cb2666457f8d177b" Nov 04 01:33:35.440031 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.439993 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" event={"ID":"7f3f33a9-ead2-4fb8-b066-5a45f35881c9","Type":"ContainerDied","Data":"1f0f6152c5c286efb0b458316c6a6b14b3f27372c2658fa7c87207431177b071"} Nov 04 01:33:35.440176 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.440099 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9" Nov 04 01:33:35.455374 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.455357 7059 scope.go:122] "RemoveContainer" containerID="bb8bfcd43e9b13d72b47ab8700ac14fc4d34a7712593d4f2cb2666457f8d177b" Nov 04 01:33:35.455703 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:33:35.455677 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb8bfcd43e9b13d72b47ab8700ac14fc4d34a7712593d4f2cb2666457f8d177b\": container with ID starting with bb8bfcd43e9b13d72b47ab8700ac14fc4d34a7712593d4f2cb2666457f8d177b not found: ID does not exist" containerID="bb8bfcd43e9b13d72b47ab8700ac14fc4d34a7712593d4f2cb2666457f8d177b" Nov 04 01:33:35.455809 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.455713 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb8bfcd43e9b13d72b47ab8700ac14fc4d34a7712593d4f2cb2666457f8d177b"} err="failed to get container status \"bb8bfcd43e9b13d72b47ab8700ac14fc4d34a7712593d4f2cb2666457f8d177b\": rpc error: code = NotFound desc = could not find container \"bb8bfcd43e9b13d72b47ab8700ac14fc4d34a7712593d4f2cb2666457f8d177b\": container with ID starting with bb8bfcd43e9b13d72b47ab8700ac14fc4d34a7712593d4f2cb2666457f8d177b not found: ID does not exist" Nov 04 01:33:35.455809 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.455736 7059 scope.go:122] "RemoveContainer" containerID="4c75c8794daef6c19284f9044940dc0b1f6d1566219ac71bb0a768ec8d11031b" Nov 04 01:33:35.475994 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.472918 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9"] Nov 04 01:33:35.478526 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.478500 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gpnp9"] Nov 04 01:33:35.486722 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.486693 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876"] Nov 04 01:33:35.491394 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:35.491373 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-nv876"] Nov 04 01:33:36.497123 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:33:36.497084 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:33:37.182134 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:37.182100 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49cff42c-6b14-409e-b5a1-9eb70c788da9" path="/var/lib/kubelet/pods/49cff42c-6b14-409e-b5a1-9eb70c788da9/volumes" Nov 04 01:33:37.182662 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:37.182646 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f3f33a9-ead2-4fb8-b066-5a45f35881c9" path="/var/lib/kubelet/pods/7f3f33a9-ead2-4fb8-b066-5a45f35881c9/volumes" Nov 04 01:33:37.960962 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:37.957201 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-920689-55545db649-5zp6c"] Nov 04 01:33:37.960962 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:37.957435 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-920689-55545db649-5zp6c" podUID="219c9bba-a9f4-4c59-a19d-8507db760298" containerName="maas-api" containerID="cri-o://45c93789f645dd46b15beebb9e3c27dcca6d478cfd5e504602e3998fa04d933a" gracePeriod=30 Nov 04 01:33:38.264436 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:38.264406 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-920689-55545db649-5zp6c" Nov 04 01:33:38.333508 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:38.333465 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/219c9bba-a9f4-4c59-a19d-8507db760298-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/219c9bba-a9f4-4c59-a19d-8507db760298-maas-api-tls\") pod \"219c9bba-a9f4-4c59-a19d-8507db760298\" (UID: \"219c9bba-a9f4-4c59-a19d-8507db760298\") " Nov 04 01:33:38.333677 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:38.333525 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/219c9bba-a9f4-4c59-a19d-8507db760298-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/219c9bba-a9f4-4c59-a19d-8507db760298-openshift-service-ca\") pod \"219c9bba-a9f4-4c59-a19d-8507db760298\" (UID: \"219c9bba-a9f4-4c59-a19d-8507db760298\") " Nov 04 01:33:38.333677 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:38.333590 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/219c9bba-a9f4-4c59-a19d-8507db760298-metrics-certs\" (UniqueName: \"kubernetes.io/secret/219c9bba-a9f4-4c59-a19d-8507db760298-metrics-certs\") pod \"219c9bba-a9f4-4c59-a19d-8507db760298\" (UID: \"219c9bba-a9f4-4c59-a19d-8507db760298\") " Nov 04 01:33:38.333876 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:38.333747 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/219c9bba-a9f4-4c59-a19d-8507db760298-kube-api-access-d9qxz\" (UniqueName: \"kubernetes.io/projected/219c9bba-a9f4-4c59-a19d-8507db760298-kube-api-access-d9qxz\") pod \"219c9bba-a9f4-4c59-a19d-8507db760298\" (UID: \"219c9bba-a9f4-4c59-a19d-8507db760298\") " Nov 04 01:33:38.335728 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:38.335694 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/219c9bba-a9f4-4c59-a19d-8507db760298-openshift-service-ca" pod "219c9bba-a9f4-4c59-a19d-8507db760298" (UID: "219c9bba-a9f4-4c59-a19d-8507db760298"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:38.338199 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:38.338109 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/219c9bba-a9f4-4c59-a19d-8507db760298-kube-api-access-d9qxz" pod "219c9bba-a9f4-4c59-a19d-8507db760298" (UID: "219c9bba-a9f4-4c59-a19d-8507db760298"). InnerVolumeSpecName "kube-api-access-d9qxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:38.354001 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:38.353966 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/219c9bba-a9f4-4c59-a19d-8507db760298-maas-api-tls" pod "219c9bba-a9f4-4c59-a19d-8507db760298" (UID: "219c9bba-a9f4-4c59-a19d-8507db760298"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:38.357046 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:38.357013 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/219c9bba-a9f4-4c59-a19d-8507db760298-metrics-certs" pod "219c9bba-a9f4-4c59-a19d-8507db760298" (UID: "219c9bba-a9f4-4c59-a19d-8507db760298"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:38.438000 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:38.437008 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/219c9bba-a9f4-4c59-a19d-8507db760298-maas-api-tls\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:38.438000 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:38.437066 7059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/219c9bba-a9f4-4c59-a19d-8507db760298-openshift-service-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:38.438000 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:38.437086 7059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/219c9bba-a9f4-4c59-a19d-8507db760298-metrics-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:38.438000 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:38.437101 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9qxz\" (UniqueName: \"kubernetes.io/projected/219c9bba-a9f4-4c59-a19d-8507db760298-kube-api-access-d9qxz\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:38.547003 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:38.546437 7059 generic.go:356] "Generic (PLEG): container finished" podID="219c9bba-a9f4-4c59-a19d-8507db760298" containerID="45c93789f645dd46b15beebb9e3c27dcca6d478cfd5e504602e3998fa04d933a" exitCode=0 Nov 04 01:33:38.547003 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:38.546669 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-920689-55545db649-5zp6c" event={"ID":"219c9bba-a9f4-4c59-a19d-8507db760298","Type":"ContainerDied","Data":"45c93789f645dd46b15beebb9e3c27dcca6d478cfd5e504602e3998fa04d933a"} Nov 04 01:33:38.547003 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:38.546727 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-920689-55545db649-5zp6c" event={"ID":"219c9bba-a9f4-4c59-a19d-8507db760298","Type":"ContainerDied","Data":"686cc0a8f6fe796a69f3be6982436a575ba8d38ef133e02c505730186f506385"} Nov 04 01:33:38.547003 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:38.546759 7059 scope.go:122] "RemoveContainer" containerID="45c93789f645dd46b15beebb9e3c27dcca6d478cfd5e504602e3998fa04d933a" Nov 04 01:33:38.547003 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:38.546944 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-920689-55545db649-5zp6c" Nov 04 01:33:38.598185 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:38.598147 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-920689-55545db649-5zp6c"] Nov 04 01:33:38.606195 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:38.606165 7059 scope.go:122] "RemoveContainer" containerID="45c93789f645dd46b15beebb9e3c27dcca6d478cfd5e504602e3998fa04d933a" Nov 04 01:33:38.606681 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:33:38.606649 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45c93789f645dd46b15beebb9e3c27dcca6d478cfd5e504602e3998fa04d933a\": container with ID starting with 45c93789f645dd46b15beebb9e3c27dcca6d478cfd5e504602e3998fa04d933a not found: ID does not exist" containerID="45c93789f645dd46b15beebb9e3c27dcca6d478cfd5e504602e3998fa04d933a" Nov 04 01:33:38.606771 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:38.606693 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45c93789f645dd46b15beebb9e3c27dcca6d478cfd5e504602e3998fa04d933a"} err="failed to get container status \"45c93789f645dd46b15beebb9e3c27dcca6d478cfd5e504602e3998fa04d933a\": rpc error: code = NotFound desc = could not find container \"45c93789f645dd46b15beebb9e3c27dcca6d478cfd5e504602e3998fa04d933a\": container with ID starting with 45c93789f645dd46b15beebb9e3c27dcca6d478cfd5e504602e3998fa04d933a not found: ID does not exist" Nov 04 01:33:38.607060 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:38.607031 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-920689-55545db649-5zp6c"] Nov 04 01:33:39.181141 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:39.181106 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="219c9bba-a9f4-4c59-a19d-8507db760298" path="/var/lib/kubelet/pods/219c9bba-a9f4-4c59-a19d-8507db760298/volumes" Nov 04 01:33:40.140634 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:40.140593 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72"] Nov 04 01:33:44.435626 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.435587 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr"] Nov 04 01:33:44.436789 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.436765 7059 cpu_manager.go:414] "containerMap: removing container" podUID="cd37362b-7164-4401-bd4b-08af565a8bc4" containerName="maas-api" Nov 04 01:33:44.436871 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.436793 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd37362b-7164-4401-bd4b-08af565a8bc4" containerName="maas-api" Nov 04 01:33:44.436871 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.436805 7059 cpu_manager.go:414] "containerMap: removing container" podUID="219c9bba-a9f4-4c59-a19d-8507db760298" containerName="maas-api" Nov 04 01:33:44.436871 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.436813 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="219c9bba-a9f4-4c59-a19d-8507db760298" containerName="maas-api" Nov 04 01:33:44.436871 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.436832 7059 cpu_manager.go:414] "containerMap: removing container" podUID="7f3f33a9-ead2-4fb8-b066-5a45f35881c9" containerName="istio-proxy" Nov 04 01:33:44.436871 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.436839 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f3f33a9-ead2-4fb8-b066-5a45f35881c9" containerName="istio-proxy" Nov 04 01:33:44.436871 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.436853 7059 cpu_manager.go:414] "containerMap: removing container" podUID="19fae239-0a50-4af6-be9b-e56c6d1bdb47" containerName="istio-proxy" Nov 04 01:33:44.436871 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.436860 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19fae239-0a50-4af6-be9b-e56c6d1bdb47" containerName="istio-proxy" Nov 04 01:33:44.436871 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.436872 7059 cpu_manager.go:414] "containerMap: removing container" podUID="fb707642-c29d-4fee-9070-06e12a582b55" containerName="maas-api" Nov 04 01:33:44.437172 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.436879 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb707642-c29d-4fee-9070-06e12a582b55" containerName="maas-api" Nov 04 01:33:44.437172 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.436894 7059 cpu_manager.go:414] "containerMap: removing container" podUID="c5e9addc-09bf-4281-a52d-916e6f15cde4" containerName="istio-proxy" Nov 04 01:33:44.437172 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.436902 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5e9addc-09bf-4281-a52d-916e6f15cde4" containerName="istio-proxy" Nov 04 01:33:44.437172 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.436944 7059 cpu_manager.go:414] "containerMap: removing container" podUID="bfeb8a23-780b-4ac0-9308-afdd49e42175" containerName="maas-api" Nov 04 01:33:44.437172 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.436952 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfeb8a23-780b-4ac0-9308-afdd49e42175" containerName="maas-api" Nov 04 01:33:44.437172 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.436976 7059 cpu_manager.go:414] "containerMap: removing container" podUID="10669ac8-e14a-4870-b1b5-f7331793d7a4" containerName="maas-api" Nov 04 01:33:44.437172 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.436983 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10669ac8-e14a-4870-b1b5-f7331793d7a4" containerName="maas-api" Nov 04 01:33:44.437172 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.437014 7059 cpu_manager.go:414] "containerMap: removing container" podUID="356d0098-8b70-42c6-bca0-31edc6a52ce9" containerName="istio-proxy" Nov 04 01:33:44.437172 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.437021 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="356d0098-8b70-42c6-bca0-31edc6a52ce9" containerName="istio-proxy" Nov 04 01:33:44.437172 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.437031 7059 cpu_manager.go:414] "containerMap: removing container" podUID="49cff42c-6b14-409e-b5a1-9eb70c788da9" containerName="istio-proxy" Nov 04 01:33:44.437172 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.437038 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49cff42c-6b14-409e-b5a1-9eb70c788da9" containerName="istio-proxy" Nov 04 01:33:44.437172 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.437061 7059 cpu_manager.go:414] "containerMap: removing container" podUID="612b8bca-a31e-4126-a517-ff449f55acd8" containerName="istio-proxy" Nov 04 01:33:44.437172 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.437068 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="612b8bca-a31e-4126-a517-ff449f55acd8" containerName="istio-proxy" Nov 04 01:33:44.437172 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.437088 7059 cpu_manager.go:414] "containerMap: removing container" podUID="b8b355cb-9a0e-45fe-8928-850062b32cdb" containerName="istio-proxy" Nov 04 01:33:44.437172 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.437095 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8b355cb-9a0e-45fe-8928-850062b32cdb" containerName="istio-proxy" Nov 04 01:33:44.437619 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.437276 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="356d0098-8b70-42c6-bca0-31edc6a52ce9" containerName="istio-proxy" Nov 04 01:33:44.437619 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.437291 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="49cff42c-6b14-409e-b5a1-9eb70c788da9" containerName="istio-proxy" Nov 04 01:33:44.437619 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.437305 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="612b8bca-a31e-4126-a517-ff449f55acd8" containerName="istio-proxy" Nov 04 01:33:44.437619 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.437314 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8b355cb-9a0e-45fe-8928-850062b32cdb" containerName="istio-proxy" Nov 04 01:33:44.437619 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.437327 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd37362b-7164-4401-bd4b-08af565a8bc4" containerName="maas-api" Nov 04 01:33:44.437619 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.437335 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="219c9bba-a9f4-4c59-a19d-8507db760298" containerName="maas-api" Nov 04 01:33:44.437619 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.437345 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f3f33a9-ead2-4fb8-b066-5a45f35881c9" containerName="istio-proxy" Nov 04 01:33:44.437619 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.437354 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="19fae239-0a50-4af6-be9b-e56c6d1bdb47" containerName="istio-proxy" Nov 04 01:33:44.437619 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.437362 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb707642-c29d-4fee-9070-06e12a582b55" containerName="maas-api" Nov 04 01:33:44.437619 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.437370 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5e9addc-09bf-4281-a52d-916e6f15cde4" containerName="istio-proxy" Nov 04 01:33:44.437619 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.437380 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfeb8a23-780b-4ac0-9308-afdd49e42175" containerName="maas-api" Nov 04 01:33:44.437619 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.437391 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="10669ac8-e14a-4870-b1b5-f7331793d7a4" containerName="maas-api" Nov 04 01:33:44.444086 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.444059 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" Nov 04 01:33:44.458298 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.458270 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr"] Nov 04 01:33:44.528122 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.528095 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c75ea1-openshift-default-dockercfg-7g542\"" Nov 04 01:33:44.552365 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.552336 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-workload-socket\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" Nov 04 01:33:44.552499 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.552377 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istio-token\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" Nov 04 01:33:44.552499 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.552489 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istio-envoy\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" Nov 04 01:33:44.552569 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.552520 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istiod-ca-cert\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" Nov 04 01:33:44.552610 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.552565 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istio-data\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" Nov 04 01:33:44.552723 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.552701 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc77n\" (UniqueName: \"kubernetes.io/projected/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-kube-api-access-dc77n\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" Nov 04 01:33:44.552857 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.552830 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-workload-certs\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" Nov 04 01:33:44.552918 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.552902 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-credential-socket\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" Nov 04 01:33:44.553044 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.553027 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istio-podinfo\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" Nov 04 01:33:44.654376 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.654348 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istio-token\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" Nov 04 01:33:44.654555 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.654403 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istio-envoy\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" Nov 04 01:33:44.654636 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.654579 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istiod-ca-cert\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" Nov 04 01:33:44.654704 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.654656 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istio-data\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" Nov 04 01:33:44.654771 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.654745 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc77n\" (UniqueName: \"kubernetes.io/projected/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-kube-api-access-dc77n\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" Nov 04 01:33:44.654858 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.654842 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-workload-certs\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" Nov 04 01:33:44.654965 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.654884 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-credential-socket\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" Nov 04 01:33:44.654965 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.654959 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istio-podinfo\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" Nov 04 01:33:44.655271 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.655096 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istio-data\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" Nov 04 01:33:44.655271 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.655138 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-workload-socket\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" Nov 04 01:33:44.655271 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.655191 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-workload-certs\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" Nov 04 01:33:44.655437 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.655283 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-credential-socket\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" Nov 04 01:33:44.655437 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.655384 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istiod-ca-cert\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" Nov 04 01:33:44.655557 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.655435 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-workload-socket\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" Nov 04 01:33:44.656969 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.656913 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istio-envoy\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" Nov 04 01:33:44.657525 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.657505 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istio-podinfo\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" Nov 04 01:33:44.662917 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.662897 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istio-token\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" Nov 04 01:33:44.663218 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.663199 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc77n\" (UniqueName: \"kubernetes.io/projected/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-kube-api-access-dc77n\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" Nov 04 01:33:44.766004 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:44.765903 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" Nov 04 01:33:45.082220 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:45.082184 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr"] Nov 04 01:33:45.082906 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:33:45.082875 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e69ed7b_5336_45e6_b0ac_ae05e4f1b79b.slice/crio-c263f8bcaf97bf8075583b82532292d40f744ff2dc063ad77e9d4b6d0207873b WatchSource:0}: Error finding container c263f8bcaf97bf8075583b82532292d40f744ff2dc063ad77e9d4b6d0207873b: Status 404 returned error can't find the container with id c263f8bcaf97bf8075583b82532292d40f744ff2dc063ad77e9d4b6d0207873b Nov 04 01:33:45.085814 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:45.085776 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:33:45.085942 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:45.085865 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:33:45.085942 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:45.085901 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:33:45.618095 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:45.618056 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" event={"ID":"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b","Type":"ContainerStarted","Data":"f42a87444dd6b08ac0692c71192185246e35612758f5caaada60428badbef3c1"} Nov 04 01:33:45.618581 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:45.618148 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" event={"ID":"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b","Type":"ContainerStarted","Data":"c263f8bcaf97bf8075583b82532292d40f744ff2dc063ad77e9d4b6d0207873b"} Nov 04 01:33:45.658211 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:45.658148 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" podStartSLOduration=1.658127072 podStartE2EDuration="1.658127072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:33:45.656834021 +0000 UTC m=+2110.598682924" watchObservedRunningTime="2026-11-04 01:33:45.658127072 +0000 UTC m=+2110.599975978" Nov 04 01:33:45.766606 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:45.766577 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" Nov 04 01:33:46.768454 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:46.768418 7059 patch_prober.go:29] interesting pod/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 04 01:33:46.768833 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:46.768477 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" podUID="4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 04 01:33:47.766862 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:47.766829 7059 patch_prober.go:29] interesting pod/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 04 01:33:47.767058 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:47.766879 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" podUID="4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 04 01:33:48.767121 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:48.767083 7059 patch_prober.go:29] interesting pod/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 04 01:33:48.767521 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:48.767136 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" podUID="4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 04 01:33:49.766986 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:49.766862 7059 patch_prober.go:29] interesting pod/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 04 01:33:49.766986 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:49.766942 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" podUID="4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 04 01:33:50.177400 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:50.177366 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr"] Nov 04 01:33:50.177861 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:50.177825 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" podUID="4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b" containerName="istio-proxy" containerID="cri-o://f42a87444dd6b08ac0692c71192185246e35612758f5caaada60428badbef3c1" gracePeriod=30 Nov 04 01:33:51.106886 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:51.106839 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl"] Nov 04 01:33:51.107391 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:51.107110 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl" podUID="a606a0bb-d6b9-4e97-863e-82c0666282bd" containerName="maas-api" containerID="cri-o://e5bcedbc0eb3db2e0572406443eece07fa6ed4a088a21622edcc7af9a280fa86" gracePeriod=30 Nov 04 01:33:51.698201 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:51.698178 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl" Nov 04 01:33:51.707567 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:51.707539 7059 generic.go:356] "Generic (PLEG): container finished" podID="a606a0bb-d6b9-4e97-863e-82c0666282bd" containerID="e5bcedbc0eb3db2e0572406443eece07fa6ed4a088a21622edcc7af9a280fa86" exitCode=0 Nov 04 01:33:51.707716 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:51.707704 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl" event={"ID":"a606a0bb-d6b9-4e97-863e-82c0666282bd","Type":"ContainerDied","Data":"e5bcedbc0eb3db2e0572406443eece07fa6ed4a088a21622edcc7af9a280fa86"} Nov 04 01:33:51.707764 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:51.707742 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl" event={"ID":"a606a0bb-d6b9-4e97-863e-82c0666282bd","Type":"ContainerDied","Data":"86ed4a982c32041f331bdfe505cbc01d72bc2aac9e48ec75807444c287609e60"} Nov 04 01:33:51.707803 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:51.707763 7059 scope.go:122] "RemoveContainer" containerID="e5bcedbc0eb3db2e0572406443eece07fa6ed4a088a21622edcc7af9a280fa86" Nov 04 01:33:51.707877 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:51.707868 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl" Nov 04 01:33:51.721765 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:51.721734 7059 scope.go:122] "RemoveContainer" containerID="e5bcedbc0eb3db2e0572406443eece07fa6ed4a088a21622edcc7af9a280fa86" Nov 04 01:33:51.722287 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:33:51.722258 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5bcedbc0eb3db2e0572406443eece07fa6ed4a088a21622edcc7af9a280fa86\": container with ID starting with e5bcedbc0eb3db2e0572406443eece07fa6ed4a088a21622edcc7af9a280fa86 not found: ID does not exist" containerID="e5bcedbc0eb3db2e0572406443eece07fa6ed4a088a21622edcc7af9a280fa86" Nov 04 01:33:51.722287 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:51.722279 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a606a0bb-d6b9-4e97-863e-82c0666282bd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a606a0bb-d6b9-4e97-863e-82c0666282bd-metrics-certs\") pod \"a606a0bb-d6b9-4e97-863e-82c0666282bd\" (UID: \"a606a0bb-d6b9-4e97-863e-82c0666282bd\") " Nov 04 01:33:51.722410 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:51.722298 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5bcedbc0eb3db2e0572406443eece07fa6ed4a088a21622edcc7af9a280fa86"} err="failed to get container status \"e5bcedbc0eb3db2e0572406443eece07fa6ed4a088a21622edcc7af9a280fa86\": rpc error: code = NotFound desc = could not find container \"e5bcedbc0eb3db2e0572406443eece07fa6ed4a088a21622edcc7af9a280fa86\": container with ID starting with e5bcedbc0eb3db2e0572406443eece07fa6ed4a088a21622edcc7af9a280fa86 not found: ID does not exist" Nov 04 01:33:51.722410 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:51.722385 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a606a0bb-d6b9-4e97-863e-82c0666282bd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a606a0bb-d6b9-4e97-863e-82c0666282bd-maas-api-tls\") pod \"a606a0bb-d6b9-4e97-863e-82c0666282bd\" (UID: \"a606a0bb-d6b9-4e97-863e-82c0666282bd\") " Nov 04 01:33:51.722484 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:51.722435 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a606a0bb-d6b9-4e97-863e-82c0666282bd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a606a0bb-d6b9-4e97-863e-82c0666282bd-openshift-service-ca\") pod \"a606a0bb-d6b9-4e97-863e-82c0666282bd\" (UID: \"a606a0bb-d6b9-4e97-863e-82c0666282bd\") " Nov 04 01:33:51.722518 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:51.722481 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a606a0bb-d6b9-4e97-863e-82c0666282bd-kube-api-access-lb6gk\" (UniqueName: \"kubernetes.io/projected/a606a0bb-d6b9-4e97-863e-82c0666282bd-kube-api-access-lb6gk\") pod \"a606a0bb-d6b9-4e97-863e-82c0666282bd\" (UID: \"a606a0bb-d6b9-4e97-863e-82c0666282bd\") " Nov 04 01:33:51.722836 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:51.722810 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a606a0bb-d6b9-4e97-863e-82c0666282bd-openshift-service-ca" pod "a606a0bb-d6b9-4e97-863e-82c0666282bd" (UID: "a606a0bb-d6b9-4e97-863e-82c0666282bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:51.723399 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:51.723378 7059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a606a0bb-d6b9-4e97-863e-82c0666282bd-openshift-service-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:51.724652 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:51.724615 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a606a0bb-d6b9-4e97-863e-82c0666282bd-kube-api-access-lb6gk" pod "a606a0bb-d6b9-4e97-863e-82c0666282bd" (UID: "a606a0bb-d6b9-4e97-863e-82c0666282bd"). InnerVolumeSpecName "kube-api-access-lb6gk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:51.725135 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:51.725101 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a606a0bb-d6b9-4e97-863e-82c0666282bd-metrics-certs" pod "a606a0bb-d6b9-4e97-863e-82c0666282bd" (UID: "a606a0bb-d6b9-4e97-863e-82c0666282bd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:51.725221 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:51.725183 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a606a0bb-d6b9-4e97-863e-82c0666282bd-maas-api-tls" pod "a606a0bb-d6b9-4e97-863e-82c0666282bd" (UID: "a606a0bb-d6b9-4e97-863e-82c0666282bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:51.836431 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:51.836391 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb6gk\" (UniqueName: \"kubernetes.io/projected/a606a0bb-d6b9-4e97-863e-82c0666282bd-kube-api-access-lb6gk\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:51.836669 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:51.836655 7059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a606a0bb-d6b9-4e97-863e-82c0666282bd-metrics-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:51.836757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:51.836747 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a606a0bb-d6b9-4e97-863e-82c0666282bd-maas-api-tls\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:51.951772 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:51.951739 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-099dd9-775897687b-7z7mn"] Nov 04 01:33:51.977896 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:51.976862 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-099dd9-5f4444cd6c-g7kl6"] Nov 04 01:33:52.063468 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:52.063437 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl"] Nov 04 01:33:52.068978 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:52.068243 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bfc54d-68ffddcbf6-htphl"] Nov 04 01:33:53.182715 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:53.182671 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a606a0bb-d6b9-4e97-863e-82c0666282bd" path="/var/lib/kubelet/pods/a606a0bb-d6b9-4e97-863e-82c0666282bd/volumes" Nov 04 01:33:55.166081 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:55.166020 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" containerID="cri-o://99883f48f1c9350ba7609da3c9ef21184e7f88e8ce0945ccb92bc3374480d24b" gracePeriod=45 Nov 04 01:33:55.521211 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:55.521175 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:33:55.618343 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:55.618311 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg"] Nov 04 01:33:55.624259 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:55.624229 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542"] Nov 04 01:33:55.651508 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:55.651470 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a747fae3-da69-424e-89c2-28fa527999e8-kube-api-access-lk4g9\" (UniqueName: \"kubernetes.io/projected/a747fae3-da69-424e-89c2-28fa527999e8-kube-api-access-lk4g9\") pod \"a747fae3-da69-424e-89c2-28fa527999e8\" (UID: \"a747fae3-da69-424e-89c2-28fa527999e8\") " Nov 04 01:33:55.651697 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:55.651629 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a747fae3-da69-424e-89c2-28fa527999e8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a747fae3-da69-424e-89c2-28fa527999e8-tmp-dir\") pod \"a747fae3-da69-424e-89c2-28fa527999e8\" (UID: \"a747fae3-da69-424e-89c2-28fa527999e8\") " Nov 04 01:33:55.651787 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:55.651773 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a747fae3-da69-424e-89c2-28fa527999e8-tls-certs\" (UniqueName: \"kubernetes.io/secret/a747fae3-da69-424e-89c2-28fa527999e8-tls-certs\") pod \"a747fae3-da69-424e-89c2-28fa527999e8\" (UID: \"a747fae3-da69-424e-89c2-28fa527999e8\") " Nov 04 01:33:55.654164 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:55.653106 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a747fae3-da69-424e-89c2-28fa527999e8-home\" (UniqueName: \"kubernetes.io/empty-dir/a747fae3-da69-424e-89c2-28fa527999e8-home\") pod \"a747fae3-da69-424e-89c2-28fa527999e8\" (UID: \"a747fae3-da69-424e-89c2-28fa527999e8\") " Nov 04 01:33:55.654164 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:55.653152 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a747fae3-da69-424e-89c2-28fa527999e8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a747fae3-da69-424e-89c2-28fa527999e8-dshm\") pod \"a747fae3-da69-424e-89c2-28fa527999e8\" (UID: \"a747fae3-da69-424e-89c2-28fa527999e8\") " Nov 04 01:33:55.654164 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:55.653197 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a747fae3-da69-424e-89c2-28fa527999e8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a747fae3-da69-424e-89c2-28fa527999e8-model-cache\") pod \"a747fae3-da69-424e-89c2-28fa527999e8\" (UID: \"a747fae3-da69-424e-89c2-28fa527999e8\") " Nov 04 01:33:55.660083 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:55.660017 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a747fae3-da69-424e-89c2-28fa527999e8-model-cache" pod "a747fae3-da69-424e-89c2-28fa527999e8" (UID: "a747fae3-da69-424e-89c2-28fa527999e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:55.660415 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:55.660386 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a747fae3-da69-424e-89c2-28fa527999e8-kube-api-access-lk4g9" pod "a747fae3-da69-424e-89c2-28fa527999e8" (UID: "a747fae3-da69-424e-89c2-28fa527999e8"). InnerVolumeSpecName "kube-api-access-lk4g9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:55.660803 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:55.660778 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a747fae3-da69-424e-89c2-28fa527999e8-tmp-dir" pod "a747fae3-da69-424e-89c2-28fa527999e8" (UID: "a747fae3-da69-424e-89c2-28fa527999e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:55.662256 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:55.662230 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a747fae3-da69-424e-89c2-28fa527999e8-home" pod "a747fae3-da69-424e-89c2-28fa527999e8" (UID: "a747fae3-da69-424e-89c2-28fa527999e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:55.664253 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:55.664207 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a747fae3-da69-424e-89c2-28fa527999e8-dshm" pod "a747fae3-da69-424e-89c2-28fa527999e8" (UID: "a747fae3-da69-424e-89c2-28fa527999e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:55.664579 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:55.664554 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a747fae3-da69-424e-89c2-28fa527999e8-tls-certs" pod "a747fae3-da69-424e-89c2-28fa527999e8" (UID: "a747fae3-da69-424e-89c2-28fa527999e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:55.754830 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:55.754646 7059 generic.go:356] "Generic (PLEG): container finished" podID="a747fae3-da69-424e-89c2-28fa527999e8" containerID="99883f48f1c9350ba7609da3c9ef21184e7f88e8ce0945ccb92bc3374480d24b" exitCode=0 Nov 04 01:33:55.755146 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:55.755002 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" event={"ID":"a747fae3-da69-424e-89c2-28fa527999e8","Type":"ContainerDied","Data":"99883f48f1c9350ba7609da3c9ef21184e7f88e8ce0945ccb92bc3374480d24b"} Nov 04 01:33:55.755146 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:55.755048 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" event={"ID":"a747fae3-da69-424e-89c2-28fa527999e8","Type":"ContainerDied","Data":"f8182e7b24dffa06ea65d33be2c554b9ccaa444cbb5fc78e736cb692b72f5c52"} Nov 04 01:33:55.755146 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:55.755072 7059 scope.go:122] "RemoveContainer" containerID="99883f48f1c9350ba7609da3c9ef21184e7f88e8ce0945ccb92bc3374480d24b" Nov 04 01:33:55.755359 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:55.755231 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72" Nov 04 01:33:55.760100 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:55.760076 7059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a747fae3-da69-424e-89c2-28fa527999e8-tmp-dir\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:55.760100 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:55.760101 7059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a747fae3-da69-424e-89c2-28fa527999e8-tls-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:55.760288 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:55.760111 7059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a747fae3-da69-424e-89c2-28fa527999e8-home\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:55.760288 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:55.760126 7059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a747fae3-da69-424e-89c2-28fa527999e8-dshm\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:55.760288 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:55.760136 7059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a747fae3-da69-424e-89c2-28fa527999e8-model-cache\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:55.760288 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:55.760146 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk4g9\" (UniqueName: \"kubernetes.io/projected/a747fae3-da69-424e-89c2-28fa527999e8-kube-api-access-lk4g9\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:55.770356 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:55.770336 7059 scope.go:122] "RemoveContainer" containerID="99883f48f1c9350ba7609da3c9ef21184e7f88e8ce0945ccb92bc3374480d24b" Nov 04 01:33:55.770651 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:33:55.770630 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99883f48f1c9350ba7609da3c9ef21184e7f88e8ce0945ccb92bc3374480d24b\": container with ID starting with 99883f48f1c9350ba7609da3c9ef21184e7f88e8ce0945ccb92bc3374480d24b not found: ID does not exist" containerID="99883f48f1c9350ba7609da3c9ef21184e7f88e8ce0945ccb92bc3374480d24b" Nov 04 01:33:55.770704 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:55.770660 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99883f48f1c9350ba7609da3c9ef21184e7f88e8ce0945ccb92bc3374480d24b"} err="failed to get container status \"99883f48f1c9350ba7609da3c9ef21184e7f88e8ce0945ccb92bc3374480d24b\": rpc error: code = NotFound desc = could not find container \"99883f48f1c9350ba7609da3c9ef21184e7f88e8ce0945ccb92bc3374480d24b\": container with ID starting with 99883f48f1c9350ba7609da3c9ef21184e7f88e8ce0945ccb92bc3374480d24b not found: ID does not exist" Nov 04 01:33:55.781505 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:55.781482 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72"] Nov 04 01:33:55.788236 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:55.788210 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-099dd9/distinct-2-w4-099dd9-kserve-6ff796867-jdb72"] Nov 04 01:33:56.952346 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:56.952307 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-099dd9-775897687b-7z7mn" podUID="53fd8082-374a-44b8-bf5d-25b56fdc7c10" containerName="payload-processing" containerID="cri-o://8e86037cc877b72234842a5c0634d757ef51f40298d36a2fb6d8f2532ab2b111" gracePeriod=25 Nov 04 01:33:56.977580 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:56.977534 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-099dd9-5f4444cd6c-g7kl6" podUID="2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d" containerName="payload-pre-processing" containerID="cri-o://5f811d7b3b328df2ce9530b84b4460a47ad2b8e2c1c08d6e8fb173f07557bcb2" gracePeriod=25 Nov 04 01:33:57.188887 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.188855 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a747fae3-da69-424e-89c2-28fa527999e8" path="/var/lib/kubelet/pods/a747fae3-da69-424e-89c2-28fa527999e8/volumes" Nov 04 01:33:57.283418 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.283122 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-099dd9-775897687b-7z7mn" Nov 04 01:33:57.293734 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.293705 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-099dd9-5f4444cd6c-g7kl6" Nov 04 01:33:57.384603 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.384563 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d-plugins-config-volume\") pod \"2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d\" (UID: \"2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d\") " Nov 04 01:33:57.384910 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.384886 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53fd8082-374a-44b8-bf5d-25b56fdc7c10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53fd8082-374a-44b8-bf5d-25b56fdc7c10-plugins-config-volume\") pod \"53fd8082-374a-44b8-bf5d-25b56fdc7c10\" (UID: \"53fd8082-374a-44b8-bf5d-25b56fdc7c10\") " Nov 04 01:33:57.385248 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.385219 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53fd8082-374a-44b8-bf5d-25b56fdc7c10-plugins-config-volume" pod "53fd8082-374a-44b8-bf5d-25b56fdc7c10" (UID: "53fd8082-374a-44b8-bf5d-25b56fdc7c10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:57.385347 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.385230 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d-kube-api-access-v5msw\" (UniqueName: \"kubernetes.io/projected/2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d-kube-api-access-v5msw\") pod \"2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d\" (UID: \"2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d\") " Nov 04 01:33:57.385347 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.385309 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53fd8082-374a-44b8-bf5d-25b56fdc7c10-kube-api-access-5fvmg\" (UniqueName: \"kubernetes.io/projected/53fd8082-374a-44b8-bf5d-25b56fdc7c10-kube-api-access-5fvmg\") pod \"53fd8082-374a-44b8-bf5d-25b56fdc7c10\" (UID: \"53fd8082-374a-44b8-bf5d-25b56fdc7c10\") " Nov 04 01:33:57.386252 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.386229 7059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53fd8082-374a-44b8-bf5d-25b56fdc7c10-plugins-config-volume\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:57.386437 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.386416 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d-plugins-config-volume" pod "2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d" (UID: "2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:57.395056 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.395028 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d-kube-api-access-v5msw" pod "2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d" (UID: "2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d"). InnerVolumeSpecName "kube-api-access-v5msw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:57.397717 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.397689 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53fd8082-374a-44b8-bf5d-25b56fdc7c10-kube-api-access-5fvmg" pod "53fd8082-374a-44b8-bf5d-25b56fdc7c10" (UID: "53fd8082-374a-44b8-bf5d-25b56fdc7c10"). InnerVolumeSpecName "kube-api-access-5fvmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:57.429208 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.429184 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" Nov 04 01:33:57.486842 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.486721 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istio-data\") pod \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " Nov 04 01:33:57.487061 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.487032 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istio-data" pod "4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b" (UID: "4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:57.487228 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.487204 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-kube-api-access-dc77n\" (UniqueName: \"kubernetes.io/projected/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-kube-api-access-dc77n\") pod \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " Nov 04 01:33:57.487339 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.487325 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istio-token\" (UniqueName: \"kubernetes.io/projected/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istio-token\") pod \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " Nov 04 01:33:57.487407 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.487386 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istiod-ca-cert\") pod \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " Nov 04 01:33:57.487524 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.487509 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-credential-socket\") pod \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " Nov 04 01:33:57.487586 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.487543 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istio-podinfo\") pod \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " Nov 04 01:33:57.487641 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.487629 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-workload-certs\") pod \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " Nov 04 01:33:57.487696 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.487672 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istio-envoy\") pod \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " Nov 04 01:33:57.487769 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.487739 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-workload-socket\") pod \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\" (UID: \"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b\") " Nov 04 01:33:57.488377 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.488350 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-credential-socket" pod "4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b" (UID: "4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:57.488476 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.488452 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5msw\" (UniqueName: \"kubernetes.io/projected/2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d-kube-api-access-v5msw\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:57.488535 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.488476 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fvmg\" (UniqueName: \"kubernetes.io/projected/53fd8082-374a-44b8-bf5d-25b56fdc7c10-kube-api-access-5fvmg\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:57.488535 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.488492 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:57.488535 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.488507 7059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d-plugins-config-volume\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:57.489260 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.489220 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-workload-certs" pod "4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b" (UID: "4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:57.489260 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.489295 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-workload-socket" pod "4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b" (UID: "4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:57.490155 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.490129 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-kube-api-access-dc77n" pod "4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b" (UID: "4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b"). InnerVolumeSpecName "kube-api-access-dc77n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:57.491516 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.491477 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istio-token" pod "4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b" (UID: "4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:57.491626 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.491602 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istio-envoy" pod "4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b" (UID: "4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:57.491871 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.491848 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istio-podinfo" pod "4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b" (UID: "4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:57.492354 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.492331 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istiod-ca-cert" pod "4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b" (UID: "4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:57.589516 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.589484 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:57.589516 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.589512 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:57.589516 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.589521 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:57.589740 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.589530 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc77n\" (UniqueName: \"kubernetes.io/projected/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-kube-api-access-dc77n\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:57.589740 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.589540 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:57.589740 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.589549 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:57.589740 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.589557 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:57.589740 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.589565 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:57.773533 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.773443 7059 generic.go:356] "Generic (PLEG): container finished" podID="4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b" containerID="f42a87444dd6b08ac0692c71192185246e35612758f5caaada60428badbef3c1" exitCode=0 Nov 04 01:33:57.773533 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.773507 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" Nov 04 01:33:57.773784 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.773530 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" event={"ID":"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b","Type":"ContainerDied","Data":"f42a87444dd6b08ac0692c71192185246e35612758f5caaada60428badbef3c1"} Nov 04 01:33:57.773784 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.773627 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr" event={"ID":"4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b","Type":"ContainerDied","Data":"c263f8bcaf97bf8075583b82532292d40f744ff2dc063ad77e9d4b6d0207873b"} Nov 04 01:33:57.773784 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.773658 7059 scope.go:122] "RemoveContainer" containerID="f42a87444dd6b08ac0692c71192185246e35612758f5caaada60428badbef3c1" Nov 04 01:33:57.775372 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.775349 7059 generic.go:356] "Generic (PLEG): container finished" podID="53fd8082-374a-44b8-bf5d-25b56fdc7c10" containerID="8e86037cc877b72234842a5c0634d757ef51f40298d36a2fb6d8f2532ab2b111" exitCode=0 Nov 04 01:33:57.775517 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.775418 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-099dd9-775897687b-7z7mn" Nov 04 01:33:57.775517 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.775439 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-099dd9-775897687b-7z7mn" event={"ID":"53fd8082-374a-44b8-bf5d-25b56fdc7c10","Type":"ContainerDied","Data":"8e86037cc877b72234842a5c0634d757ef51f40298d36a2fb6d8f2532ab2b111"} Nov 04 01:33:57.775517 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.775487 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-099dd9-775897687b-7z7mn" event={"ID":"53fd8082-374a-44b8-bf5d-25b56fdc7c10","Type":"ContainerDied","Data":"4667766d006134eb52bd983b2701f5165b7ea229f44fa52fa365d505140df591"} Nov 04 01:33:57.777307 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.777280 7059 generic.go:356] "Generic (PLEG): container finished" podID="2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d" containerID="5f811d7b3b328df2ce9530b84b4460a47ad2b8e2c1c08d6e8fb173f07557bcb2" exitCode=0 Nov 04 01:33:57.777490 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.777454 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-099dd9-5f4444cd6c-g7kl6" event={"ID":"2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d","Type":"ContainerDied","Data":"5f811d7b3b328df2ce9530b84b4460a47ad2b8e2c1c08d6e8fb173f07557bcb2"} Nov 04 01:33:57.777490 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.777505 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-099dd9-5f4444cd6c-g7kl6" event={"ID":"2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d","Type":"ContainerDied","Data":"9e961f1952e7e64a6fb34b71ae0c7cf04502779a4dbfdcdbb56dbd962927665e"} Nov 04 01:33:57.777678 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.777598 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-099dd9-5f4444cd6c-g7kl6" Nov 04 01:33:57.789947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.789915 7059 scope.go:122] "RemoveContainer" containerID="f42a87444dd6b08ac0692c71192185246e35612758f5caaada60428badbef3c1" Nov 04 01:33:57.790405 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:33:57.790278 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f42a87444dd6b08ac0692c71192185246e35612758f5caaada60428badbef3c1\": container with ID starting with f42a87444dd6b08ac0692c71192185246e35612758f5caaada60428badbef3c1 not found: ID does not exist" containerID="f42a87444dd6b08ac0692c71192185246e35612758f5caaada60428badbef3c1" Nov 04 01:33:57.790405 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.790304 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f42a87444dd6b08ac0692c71192185246e35612758f5caaada60428badbef3c1"} err="failed to get container status \"f42a87444dd6b08ac0692c71192185246e35612758f5caaada60428badbef3c1\": rpc error: code = NotFound desc = could not find container \"f42a87444dd6b08ac0692c71192185246e35612758f5caaada60428badbef3c1\": container with ID starting with f42a87444dd6b08ac0692c71192185246e35612758f5caaada60428badbef3c1 not found: ID does not exist" Nov 04 01:33:57.790405 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.790321 7059 scope.go:122] "RemoveContainer" containerID="8e86037cc877b72234842a5c0634d757ef51f40298d36a2fb6d8f2532ab2b111" Nov 04 01:33:57.801596 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.801575 7059 scope.go:122] "RemoveContainer" containerID="8e86037cc877b72234842a5c0634d757ef51f40298d36a2fb6d8f2532ab2b111" Nov 04 01:33:57.801888 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:33:57.801870 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e86037cc877b72234842a5c0634d757ef51f40298d36a2fb6d8f2532ab2b111\": container with ID starting with 8e86037cc877b72234842a5c0634d757ef51f40298d36a2fb6d8f2532ab2b111 not found: ID does not exist" containerID="8e86037cc877b72234842a5c0634d757ef51f40298d36a2fb6d8f2532ab2b111" Nov 04 01:33:57.801986 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.801893 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e86037cc877b72234842a5c0634d757ef51f40298d36a2fb6d8f2532ab2b111"} err="failed to get container status \"8e86037cc877b72234842a5c0634d757ef51f40298d36a2fb6d8f2532ab2b111\": rpc error: code = NotFound desc = could not find container \"8e86037cc877b72234842a5c0634d757ef51f40298d36a2fb6d8f2532ab2b111\": container with ID starting with 8e86037cc877b72234842a5c0634d757ef51f40298d36a2fb6d8f2532ab2b111 not found: ID does not exist" Nov 04 01:33:57.801986 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.801909 7059 scope.go:122] "RemoveContainer" containerID="5f811d7b3b328df2ce9530b84b4460a47ad2b8e2c1c08d6e8fb173f07557bcb2" Nov 04 01:33:57.812570 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.812550 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr"] Nov 04 01:33:57.813830 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.813813 7059 scope.go:122] "RemoveContainer" containerID="5f811d7b3b328df2ce9530b84b4460a47ad2b8e2c1c08d6e8fb173f07557bcb2" Nov 04 01:33:57.814167 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:33:57.814143 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f811d7b3b328df2ce9530b84b4460a47ad2b8e2c1c08d6e8fb173f07557bcb2\": container with ID starting with 5f811d7b3b328df2ce9530b84b4460a47ad2b8e2c1c08d6e8fb173f07557bcb2 not found: ID does not exist" containerID="5f811d7b3b328df2ce9530b84b4460a47ad2b8e2c1c08d6e8fb173f07557bcb2" Nov 04 01:33:57.814223 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.814179 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f811d7b3b328df2ce9530b84b4460a47ad2b8e2c1c08d6e8fb173f07557bcb2"} err="failed to get container status \"5f811d7b3b328df2ce9530b84b4460a47ad2b8e2c1c08d6e8fb173f07557bcb2\": rpc error: code = NotFound desc = could not find container \"5f811d7b3b328df2ce9530b84b4460a47ad2b8e2c1c08d6e8fb173f07557bcb2\": container with ID starting with 5f811d7b3b328df2ce9530b84b4460a47ad2b8e2c1c08d6e8fb173f07557bcb2 not found: ID does not exist" Nov 04 01:33:57.816253 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.816232 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-zsctr"] Nov 04 01:33:57.841945 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.841867 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-099dd9-5f4444cd6c-g7kl6"] Nov 04 01:33:57.853003 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.851685 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-099dd9-5f4444cd6c-g7kl6"] Nov 04 01:33:57.861424 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.861364 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-099dd9-775897687b-7z7mn"] Nov 04 01:33:57.868804 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:57.868771 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-099dd9-775897687b-7z7mn"] Nov 04 01:33:59.187764 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:59.187732 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d" path="/var/lib/kubelet/pods/2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d/volumes" Nov 04 01:33:59.188330 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:59.188312 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b" path="/var/lib/kubelet/pods/4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b/volumes" Nov 04 01:33:59.188718 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:33:59.188704 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53fd8082-374a-44b8-bf5d-25b56fdc7c10" path="/var/lib/kubelet/pods/53fd8082-374a-44b8-bf5d-25b56fdc7c10/volumes" Nov 04 01:34:05.625387 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:05.625351 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2"] Nov 04 01:34:05.626149 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:05.625629 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2" podUID="fdb4a298-c600-4362-97a0-8780f0e48532" containerName="maas-api" containerID="cri-o://e2709bd90ef774b8ed4bb31f8576064377c96d03cf2527a7eab03a9fc5bb1bff" gracePeriod=30 Nov 04 01:34:05.855997 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:05.855960 7059 generic.go:356] "Generic (PLEG): container finished" podID="fdb4a298-c600-4362-97a0-8780f0e48532" containerID="e2709bd90ef774b8ed4bb31f8576064377c96d03cf2527a7eab03a9fc5bb1bff" exitCode=0 Nov 04 01:34:05.856214 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:05.856194 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2" event={"ID":"fdb4a298-c600-4362-97a0-8780f0e48532","Type":"ContainerDied","Data":"e2709bd90ef774b8ed4bb31f8576064377c96d03cf2527a7eab03a9fc5bb1bff"} Nov 04 01:34:05.956099 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:05.956067 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2" Nov 04 01:34:06.068173 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:06.068140 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdb4a298-c600-4362-97a0-8780f0e48532-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdb4a298-c600-4362-97a0-8780f0e48532-maas-api-tls\") pod \"fdb4a298-c600-4362-97a0-8780f0e48532\" (UID: \"fdb4a298-c600-4362-97a0-8780f0e48532\") " Nov 04 01:34:06.068373 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:06.068276 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdb4a298-c600-4362-97a0-8780f0e48532-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdb4a298-c600-4362-97a0-8780f0e48532-metrics-certs\") pod \"fdb4a298-c600-4362-97a0-8780f0e48532\" (UID: \"fdb4a298-c600-4362-97a0-8780f0e48532\") " Nov 04 01:34:06.068450 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:06.068437 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdb4a298-c600-4362-97a0-8780f0e48532-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdb4a298-c600-4362-97a0-8780f0e48532-openshift-service-ca\") pod \"fdb4a298-c600-4362-97a0-8780f0e48532\" (UID: \"fdb4a298-c600-4362-97a0-8780f0e48532\") " Nov 04 01:34:06.068516 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:06.068506 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdb4a298-c600-4362-97a0-8780f0e48532-kube-api-access-xxhtm\" (UniqueName: \"kubernetes.io/projected/fdb4a298-c600-4362-97a0-8780f0e48532-kube-api-access-xxhtm\") pod \"fdb4a298-c600-4362-97a0-8780f0e48532\" (UID: \"fdb4a298-c600-4362-97a0-8780f0e48532\") " Nov 04 01:34:06.069048 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:06.069014 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdb4a298-c600-4362-97a0-8780f0e48532-openshift-service-ca" pod "fdb4a298-c600-4362-97a0-8780f0e48532" (UID: "fdb4a298-c600-4362-97a0-8780f0e48532"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:06.071094 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:06.071052 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdb4a298-c600-4362-97a0-8780f0e48532-maas-api-tls" pod "fdb4a298-c600-4362-97a0-8780f0e48532" (UID: "fdb4a298-c600-4362-97a0-8780f0e48532"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:06.072301 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:06.072277 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdb4a298-c600-4362-97a0-8780f0e48532-metrics-certs" pod "fdb4a298-c600-4362-97a0-8780f0e48532" (UID: "fdb4a298-c600-4362-97a0-8780f0e48532"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:06.072423 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:06.072330 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdb4a298-c600-4362-97a0-8780f0e48532-kube-api-access-xxhtm" pod "fdb4a298-c600-4362-97a0-8780f0e48532" (UID: "fdb4a298-c600-4362-97a0-8780f0e48532"). InnerVolumeSpecName "kube-api-access-xxhtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:06.175008 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:06.173366 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxhtm\" (UniqueName: \"kubernetes.io/projected/fdb4a298-c600-4362-97a0-8780f0e48532-kube-api-access-xxhtm\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:06.175008 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:06.173404 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdb4a298-c600-4362-97a0-8780f0e48532-maas-api-tls\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:06.175008 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:06.173418 7059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdb4a298-c600-4362-97a0-8780f0e48532-metrics-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:06.175008 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:06.173430 7059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdb4a298-c600-4362-97a0-8780f0e48532-openshift-service-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:06.516031 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:34:06.515461 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:34:06.866491 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:06.866416 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2" Nov 04 01:34:06.866491 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:06.866440 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2" event={"ID":"fdb4a298-c600-4362-97a0-8780f0e48532","Type":"ContainerDied","Data":"c61b828fa4920971e05d6d2fce0ef828e5a7b20a3013a6c737effa16940fe818"} Nov 04 01:34:06.866993 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:06.866506 7059 scope.go:122] "RemoveContainer" containerID="e2709bd90ef774b8ed4bb31f8576064377c96d03cf2527a7eab03a9fc5bb1bff" Nov 04 01:34:06.891163 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:06.891136 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2"] Nov 04 01:34:06.896626 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:06.896601 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2fff32-5b7678c588-dm9r2"] Nov 04 01:34:07.186863 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:07.186828 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdb4a298-c600-4362-97a0-8780f0e48532" path="/var/lib/kubelet/pods/fdb4a298-c600-4362-97a0-8780f0e48532/volumes" Nov 04 01:34:10.647128 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:10.647089 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" containerID="cri-o://0fc15d0302967ac7fde2be4acb9d9be123221cdcbd0f42b6fb8127b19bc6df2d" gracePeriod=45 Nov 04 01:34:10.667406 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:10.667374 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" containerID="cri-o://ad11af4294eb35acf8562e23a44958e42f0a8d839156422c0067125dcd4a1f48" gracePeriod=45 Nov 04 01:34:10.787454 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:10.787404 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": dial tcp 10.133.0.98:8000: connect: connection refused" Nov 04 01:34:10.916831 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:10.915978 7059 generic.go:356] "Generic (PLEG): container finished" podID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerID="0fc15d0302967ac7fde2be4acb9d9be123221cdcbd0f42b6fb8127b19bc6df2d" exitCode=0 Nov 04 01:34:10.916831 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:10.916168 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" event={"ID":"ef88c68d-23be-40b4-87b2-dee154f926aa","Type":"ContainerDied","Data":"0fc15d0302967ac7fde2be4acb9d9be123221cdcbd0f42b6fb8127b19bc6df2d"} Nov 04 01:34:10.919133 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:10.919100 7059 generic.go:356] "Generic (PLEG): container finished" podID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerID="ad11af4294eb35acf8562e23a44958e42f0a8d839156422c0067125dcd4a1f48" exitCode=0 Nov 04 01:34:10.919640 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:10.919279 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" event={"ID":"5c795fc5-4986-4f4d-b6f4-00baa7738286","Type":"ContainerDied","Data":"ad11af4294eb35acf8562e23a44958e42f0a8d839156422c0067125dcd4a1f48"} Nov 04 01:34:10.981118 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:10.981082 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:34:11.028677 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.028642 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:34:11.077404 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.077365 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef88c68d-23be-40b4-87b2-dee154f926aa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef88c68d-23be-40b4-87b2-dee154f926aa-tmp-dir\") pod \"ef88c68d-23be-40b4-87b2-dee154f926aa\" (UID: \"ef88c68d-23be-40b4-87b2-dee154f926aa\") " Nov 04 01:34:11.077582 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.077465 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef88c68d-23be-40b4-87b2-dee154f926aa-kube-api-access-pslhh\" (UniqueName: \"kubernetes.io/projected/ef88c68d-23be-40b4-87b2-dee154f926aa-kube-api-access-pslhh\") pod \"ef88c68d-23be-40b4-87b2-dee154f926aa\" (UID: \"ef88c68d-23be-40b4-87b2-dee154f926aa\") " Nov 04 01:34:11.077677 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.077626 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef88c68d-23be-40b4-87b2-dee154f926aa-home\" (UniqueName: \"kubernetes.io/empty-dir/ef88c68d-23be-40b4-87b2-dee154f926aa-home\") pod \"ef88c68d-23be-40b4-87b2-dee154f926aa\" (UID: \"ef88c68d-23be-40b4-87b2-dee154f926aa\") " Nov 04 01:34:11.077756 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.077716 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef88c68d-23be-40b4-87b2-dee154f926aa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef88c68d-23be-40b4-87b2-dee154f926aa-model-cache\") pod \"ef88c68d-23be-40b4-87b2-dee154f926aa\" (UID: \"ef88c68d-23be-40b4-87b2-dee154f926aa\") " Nov 04 01:34:11.077876 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.077755 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef88c68d-23be-40b4-87b2-dee154f926aa-tls-certs\" (UniqueName: \"kubernetes.io/secret/ef88c68d-23be-40b4-87b2-dee154f926aa-tls-certs\") pod \"ef88c68d-23be-40b4-87b2-dee154f926aa\" (UID: \"ef88c68d-23be-40b4-87b2-dee154f926aa\") " Nov 04 01:34:11.077876 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.077825 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef88c68d-23be-40b4-87b2-dee154f926aa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef88c68d-23be-40b4-87b2-dee154f926aa-dshm\") pod \"ef88c68d-23be-40b4-87b2-dee154f926aa\" (UID: \"ef88c68d-23be-40b4-87b2-dee154f926aa\") " Nov 04 01:34:11.083510 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.078352 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef88c68d-23be-40b4-87b2-dee154f926aa-tmp-dir" pod "ef88c68d-23be-40b4-87b2-dee154f926aa" (UID: "ef88c68d-23be-40b4-87b2-dee154f926aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:11.083510 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.078380 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef88c68d-23be-40b4-87b2-dee154f926aa-model-cache" pod "ef88c68d-23be-40b4-87b2-dee154f926aa" (UID: "ef88c68d-23be-40b4-87b2-dee154f926aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:11.083510 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.078548 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef88c68d-23be-40b4-87b2-dee154f926aa-home" pod "ef88c68d-23be-40b4-87b2-dee154f926aa" (UID: "ef88c68d-23be-40b4-87b2-dee154f926aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:11.083510 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.078575 7059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef88c68d-23be-40b4-87b2-dee154f926aa-tmp-dir\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:11.083510 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.082208 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef88c68d-23be-40b4-87b2-dee154f926aa-dshm" pod "ef88c68d-23be-40b4-87b2-dee154f926aa" (UID: "ef88c68d-23be-40b4-87b2-dee154f926aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:11.083510 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.082507 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef88c68d-23be-40b4-87b2-dee154f926aa-kube-api-access-pslhh" pod "ef88c68d-23be-40b4-87b2-dee154f926aa" (UID: "ef88c68d-23be-40b4-87b2-dee154f926aa"). InnerVolumeSpecName "kube-api-access-pslhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:11.083891 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.083863 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef88c68d-23be-40b4-87b2-dee154f926aa-tls-certs" pod "ef88c68d-23be-40b4-87b2-dee154f926aa" (UID: "ef88c68d-23be-40b4-87b2-dee154f926aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:11.179771 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.179738 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c795fc5-4986-4f4d-b6f4-00baa7738286-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c795fc5-4986-4f4d-b6f4-00baa7738286-dshm\") pod \"5c795fc5-4986-4f4d-b6f4-00baa7738286\" (UID: \"5c795fc5-4986-4f4d-b6f4-00baa7738286\") " Nov 04 01:34:11.179978 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.179838 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c795fc5-4986-4f4d-b6f4-00baa7738286-kube-api-access-4wvvb\" (UniqueName: \"kubernetes.io/projected/5c795fc5-4986-4f4d-b6f4-00baa7738286-kube-api-access-4wvvb\") pod \"5c795fc5-4986-4f4d-b6f4-00baa7738286\" (UID: \"5c795fc5-4986-4f4d-b6f4-00baa7738286\") " Nov 04 01:34:11.179978 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.179876 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c795fc5-4986-4f4d-b6f4-00baa7738286-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c795fc5-4986-4f4d-b6f4-00baa7738286-model-cache\") pod \"5c795fc5-4986-4f4d-b6f4-00baa7738286\" (UID: \"5c795fc5-4986-4f4d-b6f4-00baa7738286\") " Nov 04 01:34:11.179978 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.179910 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c795fc5-4986-4f4d-b6f4-00baa7738286-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c795fc5-4986-4f4d-b6f4-00baa7738286-tls-certs\") pod \"5c795fc5-4986-4f4d-b6f4-00baa7738286\" (UID: \"5c795fc5-4986-4f4d-b6f4-00baa7738286\") " Nov 04 01:34:11.179978 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.179961 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c795fc5-4986-4f4d-b6f4-00baa7738286-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c795fc5-4986-4f4d-b6f4-00baa7738286-tmp-dir\") pod \"5c795fc5-4986-4f4d-b6f4-00baa7738286\" (UID: \"5c795fc5-4986-4f4d-b6f4-00baa7738286\") " Nov 04 01:34:11.180168 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.180084 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c795fc5-4986-4f4d-b6f4-00baa7738286-home\" (UniqueName: \"kubernetes.io/empty-dir/5c795fc5-4986-4f4d-b6f4-00baa7738286-home\") pod \"5c795fc5-4986-4f4d-b6f4-00baa7738286\" (UID: \"5c795fc5-4986-4f4d-b6f4-00baa7738286\") " Nov 04 01:34:11.180365 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.180338 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c795fc5-4986-4f4d-b6f4-00baa7738286-tmp-dir" pod "5c795fc5-4986-4f4d-b6f4-00baa7738286" (UID: "5c795fc5-4986-4f4d-b6f4-00baa7738286"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:11.180467 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.180424 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c795fc5-4986-4f4d-b6f4-00baa7738286-model-cache" pod "5c795fc5-4986-4f4d-b6f4-00baa7738286" (UID: "5c795fc5-4986-4f4d-b6f4-00baa7738286"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:11.180529 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.180460 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c795fc5-4986-4f4d-b6f4-00baa7738286-home" pod "5c795fc5-4986-4f4d-b6f4-00baa7738286" (UID: "5c795fc5-4986-4f4d-b6f4-00baa7738286"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:11.180764 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.180743 7059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c795fc5-4986-4f4d-b6f4-00baa7738286-home\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:11.180850 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.180771 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pslhh\" (UniqueName: \"kubernetes.io/projected/ef88c68d-23be-40b4-87b2-dee154f926aa-kube-api-access-pslhh\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:11.180850 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.180786 7059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef88c68d-23be-40b4-87b2-dee154f926aa-home\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:11.180850 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.180801 7059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef88c68d-23be-40b4-87b2-dee154f926aa-model-cache\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:11.180850 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.180816 7059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef88c68d-23be-40b4-87b2-dee154f926aa-tls-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:11.180850 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.180829 7059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef88c68d-23be-40b4-87b2-dee154f926aa-dshm\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:11.180850 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.180843 7059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c795fc5-4986-4f4d-b6f4-00baa7738286-model-cache\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:11.181088 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.180857 7059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c795fc5-4986-4f4d-b6f4-00baa7738286-tmp-dir\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:11.182781 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.182757 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c795fc5-4986-4f4d-b6f4-00baa7738286-kube-api-access-4wvvb" pod "5c795fc5-4986-4f4d-b6f4-00baa7738286" (UID: "5c795fc5-4986-4f4d-b6f4-00baa7738286"). InnerVolumeSpecName "kube-api-access-4wvvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:11.183662 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.183643 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c795fc5-4986-4f4d-b6f4-00baa7738286-tls-certs" pod "5c795fc5-4986-4f4d-b6f4-00baa7738286" (UID: "5c795fc5-4986-4f4d-b6f4-00baa7738286"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:11.184027 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.184010 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c795fc5-4986-4f4d-b6f4-00baa7738286-dshm" pod "5c795fc5-4986-4f4d-b6f4-00baa7738286" (UID: "5c795fc5-4986-4f4d-b6f4-00baa7738286"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:11.282343 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.282314 7059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c795fc5-4986-4f4d-b6f4-00baa7738286-dshm\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:11.282343 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.282343 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wvvb\" (UniqueName: \"kubernetes.io/projected/5c795fc5-4986-4f4d-b6f4-00baa7738286-kube-api-access-4wvvb\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:11.282540 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.282354 7059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c795fc5-4986-4f4d-b6f4-00baa7738286-tls-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:11.826758 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.826666 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb"] Nov 04 01:34:11.827206 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.826903 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb" podUID="ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7" containerName="maas-api" containerID="cri-o://ba2c0c5c88b4526fc1c25203effa81a548b653ccf8cf8b057e96aad5788cec3b" gracePeriod=30 Nov 04 01:34:11.930184 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.930157 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" Nov 04 01:34:11.930184 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.930171 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg" event={"ID":"ef88c68d-23be-40b4-87b2-dee154f926aa","Type":"ContainerDied","Data":"028b4f0fc428b3edbd6da5b6ec0a2aef670c68305b56e18136ed1b4027ebaf55"} Nov 04 01:34:11.930443 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.930227 7059 scope.go:122] "RemoveContainer" containerID="0fc15d0302967ac7fde2be4acb9d9be123221cdcbd0f42b6fb8127b19bc6df2d" Nov 04 01:34:11.933081 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.933052 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" event={"ID":"5c795fc5-4986-4f4d-b6f4-00baa7738286","Type":"ContainerDied","Data":"7de26b13e822c694d2f0881e92a73581c180f03fb4d52bf53e1f2be185201ec7"} Nov 04 01:34:11.933208 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.933080 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" Nov 04 01:34:11.944048 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.944026 7059 scope.go:122] "RemoveContainer" containerID="ad11af4294eb35acf8562e23a44958e42f0a8d839156422c0067125dcd4a1f48" Nov 04 01:34:11.957067 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.957036 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg"] Nov 04 01:34:11.958423 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.958390 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:34:11.963564 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.963482 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-9c8bf5/distinct-2-w5-9c8bf5-kserve-55dd4c9d48-bmfhg"] Nov 04 01:34:11.985025 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.984986 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542"] Nov 04 01:34:11.990916 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:11.990882 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-9c8bf5/distinct-w5-9c8bf5-kserve-5df7696bbc-cg542"] Nov 04 01:34:12.334776 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:12.334748 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb" Nov 04 01:34:12.432157 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:12.432129 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7-kube-api-access-vj8lw\" (UniqueName: \"kubernetes.io/projected/ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7-kube-api-access-vj8lw\") pod \"ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7\" (UID: \"ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7\") " Nov 04 01:34:12.432300 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:12.432217 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7-maas-api-tls\") pod \"ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7\" (UID: \"ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7\") " Nov 04 01:34:12.432373 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:12.432358 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7-metrics-certs\") pod \"ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7\" (UID: \"ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7\") " Nov 04 01:34:12.432503 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:12.432487 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7-openshift-service-ca\") pod \"ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7\" (UID: \"ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7\") " Nov 04 01:34:12.433089 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:12.433056 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7-openshift-service-ca" pod "ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7" (UID: "ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:12.434448 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:12.434428 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7-maas-api-tls" pod "ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7" (UID: "ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:12.434995 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:12.434972 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7-metrics-certs" pod "ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7" (UID: "ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:12.435084 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:12.435019 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7-kube-api-access-vj8lw" pod "ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7" (UID: "ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7"). InnerVolumeSpecName "kube-api-access-vj8lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:12.534112 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:12.534077 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7-maas-api-tls\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:12.534112 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:12.534116 7059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7-metrics-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:12.534350 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:12.534130 7059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7-openshift-service-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:12.534350 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:12.534145 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj8lw\" (UniqueName: \"kubernetes.io/projected/ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7-kube-api-access-vj8lw\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:12.727101 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:12.727067 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-9c8bf5-646c6d9774-4862q"] Nov 04 01:34:12.754522 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:12.754482 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-9c8bf5-9758b5cf-9hsn2"] Nov 04 01:34:12.944847 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:12.944809 7059 generic.go:356] "Generic (PLEG): container finished" podID="ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7" containerID="ba2c0c5c88b4526fc1c25203effa81a548b653ccf8cf8b057e96aad5788cec3b" exitCode=0 Nov 04 01:34:12.945386 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:12.944859 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb" Nov 04 01:34:12.945386 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:12.944902 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb" event={"ID":"ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7","Type":"ContainerDied","Data":"ba2c0c5c88b4526fc1c25203effa81a548b653ccf8cf8b057e96aad5788cec3b"} Nov 04 01:34:12.945386 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:12.945003 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb" event={"ID":"ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7","Type":"ContainerDied","Data":"489de3e40f8fddcaaef49147c81a4e614d19d6cbb1e8c910f0870cdc2a930ebb"} Nov 04 01:34:12.945386 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:12.945035 7059 scope.go:122] "RemoveContainer" containerID="ba2c0c5c88b4526fc1c25203effa81a548b653ccf8cf8b057e96aad5788cec3b" Nov 04 01:34:12.958223 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:12.958205 7059 scope.go:122] "RemoveContainer" containerID="ba2c0c5c88b4526fc1c25203effa81a548b653ccf8cf8b057e96aad5788cec3b" Nov 04 01:34:12.958514 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:34:12.958490 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba2c0c5c88b4526fc1c25203effa81a548b653ccf8cf8b057e96aad5788cec3b\": container with ID starting with ba2c0c5c88b4526fc1c25203effa81a548b653ccf8cf8b057e96aad5788cec3b not found: ID does not exist" containerID="ba2c0c5c88b4526fc1c25203effa81a548b653ccf8cf8b057e96aad5788cec3b" Nov 04 01:34:12.958581 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:12.958526 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba2c0c5c88b4526fc1c25203effa81a548b653ccf8cf8b057e96aad5788cec3b"} err="failed to get container status \"ba2c0c5c88b4526fc1c25203effa81a548b653ccf8cf8b057e96aad5788cec3b\": rpc error: code = NotFound desc = could not find container \"ba2c0c5c88b4526fc1c25203effa81a548b653ccf8cf8b057e96aad5788cec3b\": container with ID starting with ba2c0c5c88b4526fc1c25203effa81a548b653ccf8cf8b057e96aad5788cec3b not found: ID does not exist" Nov 04 01:34:12.978526 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:12.978469 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb"] Nov 04 01:34:12.983876 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:12.983851 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-099dd9-858f7d7ffd-l2qxb"] Nov 04 01:34:13.221836 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:13.221787 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" path="/var/lib/kubelet/pods/5c795fc5-4986-4f4d-b6f4-00baa7738286/volumes" Nov 04 01:34:13.222714 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:13.222680 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7" path="/var/lib/kubelet/pods/ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7/volumes" Nov 04 01:34:13.223374 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:13.223353 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" path="/var/lib/kubelet/pods/ef88c68d-23be-40b4-87b2-dee154f926aa/volumes" Nov 04 01:34:15.475453 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:15.475416 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz"] Nov 04 01:34:15.494377 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:15.494349 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z"] Nov 04 01:34:17.731266 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:17.730510 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-9c8bf5-646c6d9774-4862q" podUID="158365b9-7071-4a6d-b2a4-f6b56d795ec1" containerName="payload-processing" containerID="cri-o://635f93a606a4c10f2fe4d9f15a86fa07dcd9fabfeb5b131212eac77cfc4095d9" gracePeriod=25 Nov 04 01:34:17.758066 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:17.757991 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9c8bf5-9758b5cf-9hsn2" podUID="142d5751-9524-4f5c-8072-d05d5ca426ad" containerName="payload-pre-processing" containerID="cri-o://cbc20e0c0e789a803cca3793bf9f27ec3d0cc9841d566664fcff950c302a7a52" gracePeriod=25 Nov 04 01:34:19.091680 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:19.091642 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2"] Nov 04 01:34:19.092141 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:19.091872 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" podUID="b59977fd-6792-40e9-9070-1f04beb3f0bf" containerName="istio-proxy" containerID="cri-o://57d4c0a099c9e69dfe379431eebf2798321cbea14f78e4720e56dc3fbcf7ac91" gracePeriod=30 Nov 04 01:34:19.117051 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:19.117023 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t"] Nov 04 01:34:19.117238 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:19.117202 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" podUID="d211e9ea-abd4-41d1-8011-e884ad33695f" containerName="istio-proxy" containerID="cri-o://cef1124da045bcb53fed0343774e1cbc8920bf4566d132b9753d8575b065ff18" gracePeriod=30 Nov 04 01:34:20.515265 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:20.515224 7059 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:34:20.515696 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:20.515316 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" podUID="b59977fd-6792-40e9-9070-1f04beb3f0bf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:34:21.195961 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:21.195911 7059 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:34:21.196167 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:21.195984 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" podUID="d211e9ea-abd4-41d1-8011-e884ad33695f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:34:21.573875 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:21.573788 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-586541-6bdd6489cf-pn6dk"] Nov 04 01:34:24.345903 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.345671 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-9c8bf5-646c6d9774-4862q" podUID="158365b9-7071-4a6d-b2a4-f6b56d795ec1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:34:24.347572 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.347533 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9c8bf5-9758b5cf-9hsn2" podUID="142d5751-9524-4f5c-8072-d05d5ca426ad" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:34:24.408652 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.408626 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:34:24.452627 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.452603 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:34:24.540405 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.540366 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-credential-socket\") pod \"b59977fd-6792-40e9-9070-1f04beb3f0bf\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " Nov 04 01:34:24.540591 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.540412 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-workload-socket\") pod \"d211e9ea-abd4-41d1-8011-e884ad33695f\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " Nov 04 01:34:24.540591 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.540473 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b59977fd-6792-40e9-9070-1f04beb3f0bf-istio-token\" (UniqueName: \"kubernetes.io/projected/b59977fd-6792-40e9-9070-1f04beb3f0bf-istio-token\") pod \"b59977fd-6792-40e9-9070-1f04beb3f0bf\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " Nov 04 01:34:24.540591 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.540523 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d211e9ea-abd4-41d1-8011-e884ad33695f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d211e9ea-abd4-41d1-8011-e884ad33695f-istiod-ca-cert\") pod \"d211e9ea-abd4-41d1-8011-e884ad33695f\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " Nov 04 01:34:24.540591 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.540559 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-istio-data\") pod \"b59977fd-6792-40e9-9070-1f04beb3f0bf\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " Nov 04 01:34:24.540983 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.540625 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-workload-certs\") pod \"b59977fd-6792-40e9-9070-1f04beb3f0bf\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " Nov 04 01:34:24.540983 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.540678 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-istio-envoy\") pod \"d211e9ea-abd4-41d1-8011-e884ad33695f\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " Nov 04 01:34:24.540983 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.540741 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-credential-socket" pod "b59977fd-6792-40e9-9070-1f04beb3f0bf" (UID: "b59977fd-6792-40e9-9070-1f04beb3f0bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:24.540983 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.540818 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-workload-socket" pod "d211e9ea-abd4-41d1-8011-e884ad33695f" (UID: "d211e9ea-abd4-41d1-8011-e884ad33695f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:24.544565 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.541043 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d211e9ea-abd4-41d1-8011-e884ad33695f-istio-token\" (UniqueName: \"kubernetes.io/projected/d211e9ea-abd4-41d1-8011-e884ad33695f-istio-token\") pod \"d211e9ea-abd4-41d1-8011-e884ad33695f\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " Nov 04 01:34:24.544565 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.541140 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-istio-data" pod "b59977fd-6792-40e9-9070-1f04beb3f0bf" (UID: "b59977fd-6792-40e9-9070-1f04beb3f0bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:24.544565 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.541197 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-workload-certs" pod "b59977fd-6792-40e9-9070-1f04beb3f0bf" (UID: "b59977fd-6792-40e9-9070-1f04beb3f0bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:24.544565 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.541209 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-credential-socket\") pod \"d211e9ea-abd4-41d1-8011-e884ad33695f\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " Nov 04 01:34:24.544565 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.541326 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d211e9ea-abd4-41d1-8011-e884ad33695f-kube-api-access-7tmb6\" (UniqueName: \"kubernetes.io/projected/d211e9ea-abd4-41d1-8011-e884ad33695f-kube-api-access-7tmb6\") pod \"d211e9ea-abd4-41d1-8011-e884ad33695f\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " Nov 04 01:34:24.544565 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.541338 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d211e9ea-abd4-41d1-8011-e884ad33695f-istiod-ca-cert" pod "d211e9ea-abd4-41d1-8011-e884ad33695f" (UID: "d211e9ea-abd4-41d1-8011-e884ad33695f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:24.544565 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.541414 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d211e9ea-abd4-41d1-8011-e884ad33695f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d211e9ea-abd4-41d1-8011-e884ad33695f-istio-podinfo\") pod \"d211e9ea-abd4-41d1-8011-e884ad33695f\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " Nov 04 01:34:24.544565 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.541447 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-credential-socket" pod "d211e9ea-abd4-41d1-8011-e884ad33695f" (UID: "d211e9ea-abd4-41d1-8011-e884ad33695f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:24.544565 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.541531 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-workload-certs\") pod \"d211e9ea-abd4-41d1-8011-e884ad33695f\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " Nov 04 01:34:24.544565 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.541588 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b59977fd-6792-40e9-9070-1f04beb3f0bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b59977fd-6792-40e9-9070-1f04beb3f0bf-istio-podinfo\") pod \"b59977fd-6792-40e9-9070-1f04beb3f0bf\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " Nov 04 01:34:24.544565 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.541644 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-istio-data\") pod \"d211e9ea-abd4-41d1-8011-e884ad33695f\" (UID: \"d211e9ea-abd4-41d1-8011-e884ad33695f\") " Nov 04 01:34:24.544565 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.541715 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-workload-certs" pod "d211e9ea-abd4-41d1-8011-e884ad33695f" (UID: "d211e9ea-abd4-41d1-8011-e884ad33695f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:24.544565 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.541721 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-istio-envoy\") pod \"b59977fd-6792-40e9-9070-1f04beb3f0bf\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " Nov 04 01:34:24.544565 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.541984 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b59977fd-6792-40e9-9070-1f04beb3f0bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b59977fd-6792-40e9-9070-1f04beb3f0bf-istiod-ca-cert\") pod \"b59977fd-6792-40e9-9070-1f04beb3f0bf\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " Nov 04 01:34:24.544565 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.542048 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-workload-socket\") pod \"b59977fd-6792-40e9-9070-1f04beb3f0bf\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " Nov 04 01:34:24.544565 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.542076 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b59977fd-6792-40e9-9070-1f04beb3f0bf-kube-api-access-4bwrs\" (UniqueName: \"kubernetes.io/projected/b59977fd-6792-40e9-9070-1f04beb3f0bf-kube-api-access-4bwrs\") pod \"b59977fd-6792-40e9-9070-1f04beb3f0bf\" (UID: \"b59977fd-6792-40e9-9070-1f04beb3f0bf\") " Nov 04 01:34:24.544565 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.542940 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b59977fd-6792-40e9-9070-1f04beb3f0bf-istiod-ca-cert" pod "b59977fd-6792-40e9-9070-1f04beb3f0bf" (UID: "b59977fd-6792-40e9-9070-1f04beb3f0bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:24.544565 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.543671 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.544565 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.543698 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b59977fd-6792-40e9-9070-1f04beb3f0bf-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.544565 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.543713 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.544565 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.543728 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.544565 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.543757 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d211e9ea-abd4-41d1-8011-e884ad33695f-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.544565 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.543770 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.544565 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.543783 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.544565 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.543796 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.544565 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.544000 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-workload-socket" pod "b59977fd-6792-40e9-9070-1f04beb3f0bf" (UID: "b59977fd-6792-40e9-9070-1f04beb3f0bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:24.544565 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.544303 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d211e9ea-abd4-41d1-8011-e884ad33695f-istio-token" pod "d211e9ea-abd4-41d1-8011-e884ad33695f" (UID: "d211e9ea-abd4-41d1-8011-e884ad33695f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:24.546462 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.545783 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-istio-envoy" pod "b59977fd-6792-40e9-9070-1f04beb3f0bf" (UID: "b59977fd-6792-40e9-9070-1f04beb3f0bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:24.546462 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.546113 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-istio-envoy" pod "d211e9ea-abd4-41d1-8011-e884ad33695f" (UID: "d211e9ea-abd4-41d1-8011-e884ad33695f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:24.548002 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.547676 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-istio-data" pod "d211e9ea-abd4-41d1-8011-e884ad33695f" (UID: "d211e9ea-abd4-41d1-8011-e884ad33695f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:24.548881 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.548323 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b59977fd-6792-40e9-9070-1f04beb3f0bf-kube-api-access-4bwrs" pod "b59977fd-6792-40e9-9070-1f04beb3f0bf" (UID: "b59977fd-6792-40e9-9070-1f04beb3f0bf"). InnerVolumeSpecName "kube-api-access-4bwrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:24.548881 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.548602 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b59977fd-6792-40e9-9070-1f04beb3f0bf-istio-token" pod "b59977fd-6792-40e9-9070-1f04beb3f0bf" (UID: "b59977fd-6792-40e9-9070-1f04beb3f0bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:24.548881 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.548663 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b59977fd-6792-40e9-9070-1f04beb3f0bf-istio-podinfo" pod "b59977fd-6792-40e9-9070-1f04beb3f0bf" (UID: "b59977fd-6792-40e9-9070-1f04beb3f0bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:34:24.548881 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.548712 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d211e9ea-abd4-41d1-8011-e884ad33695f-istio-podinfo" pod "d211e9ea-abd4-41d1-8011-e884ad33695f" (UID: "d211e9ea-abd4-41d1-8011-e884ad33695f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:34:24.550396 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.550369 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d211e9ea-abd4-41d1-8011-e884ad33695f-kube-api-access-7tmb6" pod "d211e9ea-abd4-41d1-8011-e884ad33695f" (UID: "d211e9ea-abd4-41d1-8011-e884ad33695f"). InnerVolumeSpecName "kube-api-access-7tmb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:24.644801 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.644729 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b59977fd-6792-40e9-9070-1f04beb3f0bf-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.644801 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.644758 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.644801 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.644768 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.644801 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.644777 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b59977fd-6792-40e9-9070-1f04beb3f0bf-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.644801 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.644786 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bwrs\" (UniqueName: \"kubernetes.io/projected/b59977fd-6792-40e9-9070-1f04beb3f0bf-kube-api-access-4bwrs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.644801 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.644796 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b59977fd-6792-40e9-9070-1f04beb3f0bf-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.644801 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.644804 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d211e9ea-abd4-41d1-8011-e884ad33695f-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.645143 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.644812 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d211e9ea-abd4-41d1-8011-e884ad33695f-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.645143 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.644820 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tmb6\" (UniqueName: \"kubernetes.io/projected/d211e9ea-abd4-41d1-8011-e884ad33695f-kube-api-access-7tmb6\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.645143 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:24.644829 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d211e9ea-abd4-41d1-8011-e884ad33695f-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:25.141320 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:25.141287 7059 generic.go:356] "Generic (PLEG): container finished" podID="b59977fd-6792-40e9-9070-1f04beb3f0bf" containerID="57d4c0a099c9e69dfe379431eebf2798321cbea14f78e4720e56dc3fbcf7ac91" exitCode=0 Nov 04 01:34:25.141481 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:25.141333 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" Nov 04 01:34:25.141481 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:25.141360 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" event={"ID":"b59977fd-6792-40e9-9070-1f04beb3f0bf","Type":"ContainerDied","Data":"57d4c0a099c9e69dfe379431eebf2798321cbea14f78e4720e56dc3fbcf7ac91"} Nov 04 01:34:25.141481 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:25.141412 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2" event={"ID":"b59977fd-6792-40e9-9070-1f04beb3f0bf","Type":"ContainerDied","Data":"0371cf25dee10ab30827bf7f9ee0e23280cd20f51b93c4dcd0b8e1df6cdfc099"} Nov 04 01:34:25.141481 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:25.141435 7059 scope.go:122] "RemoveContainer" containerID="57d4c0a099c9e69dfe379431eebf2798321cbea14f78e4720e56dc3fbcf7ac91" Nov 04 01:34:25.147966 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:25.143022 7059 generic.go:356] "Generic (PLEG): container finished" podID="d211e9ea-abd4-41d1-8011-e884ad33695f" containerID="cef1124da045bcb53fed0343774e1cbc8920bf4566d132b9753d8575b065ff18" exitCode=0 Nov 04 01:34:25.147966 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:25.143082 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" Nov 04 01:34:25.147966 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:25.143159 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" event={"ID":"d211e9ea-abd4-41d1-8011-e884ad33695f","Type":"ContainerDied","Data":"cef1124da045bcb53fed0343774e1cbc8920bf4566d132b9753d8575b065ff18"} Nov 04 01:34:25.147966 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:25.143215 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t" event={"ID":"d211e9ea-abd4-41d1-8011-e884ad33695f","Type":"ContainerDied","Data":"5b1f5087e7910a8d7912afeef61069391b716a980ddc9c0089f156d6d73d07cb"} Nov 04 01:34:25.153356 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:25.153340 7059 scope.go:122] "RemoveContainer" containerID="57d4c0a099c9e69dfe379431eebf2798321cbea14f78e4720e56dc3fbcf7ac91" Nov 04 01:34:25.153664 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:34:25.153635 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57d4c0a099c9e69dfe379431eebf2798321cbea14f78e4720e56dc3fbcf7ac91\": container with ID starting with 57d4c0a099c9e69dfe379431eebf2798321cbea14f78e4720e56dc3fbcf7ac91 not found: ID does not exist" containerID="57d4c0a099c9e69dfe379431eebf2798321cbea14f78e4720e56dc3fbcf7ac91" Nov 04 01:34:25.153725 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:25.153672 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57d4c0a099c9e69dfe379431eebf2798321cbea14f78e4720e56dc3fbcf7ac91"} err="failed to get container status \"57d4c0a099c9e69dfe379431eebf2798321cbea14f78e4720e56dc3fbcf7ac91\": rpc error: code = NotFound desc = could not find container \"57d4c0a099c9e69dfe379431eebf2798321cbea14f78e4720e56dc3fbcf7ac91\": container with ID starting with 57d4c0a099c9e69dfe379431eebf2798321cbea14f78e4720e56dc3fbcf7ac91 not found: ID does not exist" Nov 04 01:34:25.153725 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:25.153690 7059 scope.go:122] "RemoveContainer" containerID="cef1124da045bcb53fed0343774e1cbc8920bf4566d132b9753d8575b065ff18" Nov 04 01:34:25.166986 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:25.166969 7059 scope.go:122] "RemoveContainer" containerID="cef1124da045bcb53fed0343774e1cbc8920bf4566d132b9753d8575b065ff18" Nov 04 01:34:25.167279 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:34:25.167257 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cef1124da045bcb53fed0343774e1cbc8920bf4566d132b9753d8575b065ff18\": container with ID starting with cef1124da045bcb53fed0343774e1cbc8920bf4566d132b9753d8575b065ff18 not found: ID does not exist" containerID="cef1124da045bcb53fed0343774e1cbc8920bf4566d132b9753d8575b065ff18" Nov 04 01:34:25.167328 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:25.167288 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cef1124da045bcb53fed0343774e1cbc8920bf4566d132b9753d8575b065ff18"} err="failed to get container status \"cef1124da045bcb53fed0343774e1cbc8920bf4566d132b9753d8575b065ff18\": rpc error: code = NotFound desc = could not find container \"cef1124da045bcb53fed0343774e1cbc8920bf4566d132b9753d8575b065ff18\": container with ID starting with cef1124da045bcb53fed0343774e1cbc8920bf4566d132b9753d8575b065ff18 not found: ID does not exist" Nov 04 01:34:25.181132 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:25.181110 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t"] Nov 04 01:34:25.181327 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:25.181308 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-6t86t"] Nov 04 01:34:25.193147 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:25.193122 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2"] Nov 04 01:34:25.197361 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:25.197340 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-vlkm2"] Nov 04 01:34:26.574357 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:26.574315 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-586541-6bdd6489cf-pn6dk" podUID="b3ea0000-873e-4789-8ad1-d182a29d4489" containerName="payload-pre-processing" containerID="cri-o://5d8b7ecc7b4fdc5d5334c45804632a8719cc77ea8c3bf203baa05165cc5a6f76" gracePeriod=25 Nov 04 01:34:26.942967 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:26.942941 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-586541-6bdd6489cf-pn6dk" Nov 04 01:34:26.992219 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:26.992188 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3ea0000-873e-4789-8ad1-d182a29d4489-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ea0000-873e-4789-8ad1-d182a29d4489-plugins-config-volume\") pod \"b3ea0000-873e-4789-8ad1-d182a29d4489\" (UID: \"b3ea0000-873e-4789-8ad1-d182a29d4489\") " Nov 04 01:34:26.992554 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:26.992527 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3ea0000-873e-4789-8ad1-d182a29d4489-kube-api-access-sgbmg\" (UniqueName: \"kubernetes.io/projected/b3ea0000-873e-4789-8ad1-d182a29d4489-kube-api-access-sgbmg\") pod \"b3ea0000-873e-4789-8ad1-d182a29d4489\" (UID: \"b3ea0000-873e-4789-8ad1-d182a29d4489\") " Nov 04 01:34:26.993379 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:26.993348 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3ea0000-873e-4789-8ad1-d182a29d4489-plugins-config-volume" pod "b3ea0000-873e-4789-8ad1-d182a29d4489" (UID: "b3ea0000-873e-4789-8ad1-d182a29d4489"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:26.997014 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:26.996985 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ea0000-873e-4789-8ad1-d182a29d4489-kube-api-access-sgbmg" pod "b3ea0000-873e-4789-8ad1-d182a29d4489" (UID: "b3ea0000-873e-4789-8ad1-d182a29d4489"). InnerVolumeSpecName "kube-api-access-sgbmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:27.102024 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:27.101883 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgbmg\" (UniqueName: \"kubernetes.io/projected/b3ea0000-873e-4789-8ad1-d182a29d4489-kube-api-access-sgbmg\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:27.102263 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:27.102239 7059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ea0000-873e-4789-8ad1-d182a29d4489-plugins-config-volume\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:27.166629 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:27.166550 7059 generic.go:356] "Generic (PLEG): container finished" podID="b3ea0000-873e-4789-8ad1-d182a29d4489" containerID="5d8b7ecc7b4fdc5d5334c45804632a8719cc77ea8c3bf203baa05165cc5a6f76" exitCode=0 Nov 04 01:34:27.166884 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:27.166596 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-586541-6bdd6489cf-pn6dk" event={"ID":"b3ea0000-873e-4789-8ad1-d182a29d4489","Type":"ContainerDied","Data":"5d8b7ecc7b4fdc5d5334c45804632a8719cc77ea8c3bf203baa05165cc5a6f76"} Nov 04 01:34:27.166884 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:27.166685 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-586541-6bdd6489cf-pn6dk" event={"ID":"b3ea0000-873e-4789-8ad1-d182a29d4489","Type":"ContainerDied","Data":"8906446ddcc2ae0407d9527acca1ee748764b03882e2022a89c08c728c1b016b"} Nov 04 01:34:27.166884 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:27.166715 7059 scope.go:122] "RemoveContainer" containerID="5d8b7ecc7b4fdc5d5334c45804632a8719cc77ea8c3bf203baa05165cc5a6f76" Nov 04 01:34:27.166884 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:27.166751 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-586541-6bdd6489cf-pn6dk" Nov 04 01:34:27.179109 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:27.179082 7059 scope.go:122] "RemoveContainer" containerID="5d8b7ecc7b4fdc5d5334c45804632a8719cc77ea8c3bf203baa05165cc5a6f76" Nov 04 01:34:27.179414 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:34:27.179392 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d8b7ecc7b4fdc5d5334c45804632a8719cc77ea8c3bf203baa05165cc5a6f76\": container with ID starting with 5d8b7ecc7b4fdc5d5334c45804632a8719cc77ea8c3bf203baa05165cc5a6f76 not found: ID does not exist" containerID="5d8b7ecc7b4fdc5d5334c45804632a8719cc77ea8c3bf203baa05165cc5a6f76" Nov 04 01:34:27.179509 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:27.179419 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d8b7ecc7b4fdc5d5334c45804632a8719cc77ea8c3bf203baa05165cc5a6f76"} err="failed to get container status \"5d8b7ecc7b4fdc5d5334c45804632a8719cc77ea8c3bf203baa05165cc5a6f76\": rpc error: code = NotFound desc = could not find container \"5d8b7ecc7b4fdc5d5334c45804632a8719cc77ea8c3bf203baa05165cc5a6f76\": container with ID starting with 5d8b7ecc7b4fdc5d5334c45804632a8719cc77ea8c3bf203baa05165cc5a6f76 not found: ID does not exist" Nov 04 01:34:27.181488 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:27.181466 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b59977fd-6792-40e9-9070-1f04beb3f0bf" path="/var/lib/kubelet/pods/b59977fd-6792-40e9-9070-1f04beb3f0bf/volumes" Nov 04 01:34:27.182125 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:27.182107 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d211e9ea-abd4-41d1-8011-e884ad33695f" path="/var/lib/kubelet/pods/d211e9ea-abd4-41d1-8011-e884ad33695f/volumes" Nov 04 01:34:27.199568 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:27.199537 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-586541-6bdd6489cf-pn6dk"] Nov 04 01:34:27.202972 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:27.202948 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-586541-6bdd6489cf-pn6dk"] Nov 04 01:34:29.180472 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:29.180437 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3ea0000-873e-4789-8ad1-d182a29d4489" path="/var/lib/kubelet/pods/b3ea0000-873e-4789-8ad1-d182a29d4489/volumes" Nov 04 01:34:30.499206 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:30.499163 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" containerID="cri-o://713e989f10ddfc20e3b2695c179d01563a68536411955099c1df730b0e2b265c" gracePeriod=45 Nov 04 01:34:30.518147 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:30.518086 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" containerID="cri-o://49cb68ad09d1bf01650b6c0f82ecdd96bbd139e9032541b673d1da51560c5a09" gracePeriod=45 Nov 04 01:34:30.868979 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:30.868234 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:34:30.875742 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:30.875646 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:34:30.975801 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:30.975765 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad50a174-edcc-4898-b072-90d6c9246859-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad50a174-edcc-4898-b072-90d6c9246859-dshm\") pod \"ad50a174-edcc-4898-b072-90d6c9246859\" (UID: \"ad50a174-edcc-4898-b072-90d6c9246859\") " Nov 04 01:34:30.976011 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:30.975831 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad50a174-edcc-4898-b072-90d6c9246859-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad50a174-edcc-4898-b072-90d6c9246859-model-cache\") pod \"ad50a174-edcc-4898-b072-90d6c9246859\" (UID: \"ad50a174-edcc-4898-b072-90d6c9246859\") " Nov 04 01:34:30.976011 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:30.975912 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b51f24e-65d2-40fd-8b78-70ee1a593d00-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b51f24e-65d2-40fd-8b78-70ee1a593d00-dshm\") pod \"1b51f24e-65d2-40fd-8b78-70ee1a593d00\" (UID: \"1b51f24e-65d2-40fd-8b78-70ee1a593d00\") " Nov 04 01:34:30.976146 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:30.976101 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b51f24e-65d2-40fd-8b78-70ee1a593d00-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b51f24e-65d2-40fd-8b78-70ee1a593d00-model-cache\") pod \"1b51f24e-65d2-40fd-8b78-70ee1a593d00\" (UID: \"1b51f24e-65d2-40fd-8b78-70ee1a593d00\") " Nov 04 01:34:30.976146 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:30.976133 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad50a174-edcc-4898-b072-90d6c9246859-model-cache" pod "ad50a174-edcc-4898-b072-90d6c9246859" (UID: "ad50a174-edcc-4898-b072-90d6c9246859"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:30.976269 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:30.976172 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad50a174-edcc-4898-b072-90d6c9246859-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad50a174-edcc-4898-b072-90d6c9246859-tls-certs\") pod \"ad50a174-edcc-4898-b072-90d6c9246859\" (UID: \"ad50a174-edcc-4898-b072-90d6c9246859\") " Nov 04 01:34:30.976269 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:30.976203 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b51f24e-65d2-40fd-8b78-70ee1a593d00-kube-api-access-whbzr\" (UniqueName: \"kubernetes.io/projected/1b51f24e-65d2-40fd-8b78-70ee1a593d00-kube-api-access-whbzr\") pod \"1b51f24e-65d2-40fd-8b78-70ee1a593d00\" (UID: \"1b51f24e-65d2-40fd-8b78-70ee1a593d00\") " Nov 04 01:34:30.976269 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:30.976242 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b51f24e-65d2-40fd-8b78-70ee1a593d00-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b51f24e-65d2-40fd-8b78-70ee1a593d00-tmp-dir\") pod \"1b51f24e-65d2-40fd-8b78-70ee1a593d00\" (UID: \"1b51f24e-65d2-40fd-8b78-70ee1a593d00\") " Nov 04 01:34:30.976428 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:30.976317 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b51f24e-65d2-40fd-8b78-70ee1a593d00-home\" (UniqueName: \"kubernetes.io/empty-dir/1b51f24e-65d2-40fd-8b78-70ee1a593d00-home\") pod \"1b51f24e-65d2-40fd-8b78-70ee1a593d00\" (UID: \"1b51f24e-65d2-40fd-8b78-70ee1a593d00\") " Nov 04 01:34:30.976428 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:30.976339 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b51f24e-65d2-40fd-8b78-70ee1a593d00-model-cache" pod "1b51f24e-65d2-40fd-8b78-70ee1a593d00" (UID: "1b51f24e-65d2-40fd-8b78-70ee1a593d00"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:30.976428 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:30.976360 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad50a174-edcc-4898-b072-90d6c9246859-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad50a174-edcc-4898-b072-90d6c9246859-tmp-dir\") pod \"ad50a174-edcc-4898-b072-90d6c9246859\" (UID: \"ad50a174-edcc-4898-b072-90d6c9246859\") " Nov 04 01:34:30.976604 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:30.976433 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad50a174-edcc-4898-b072-90d6c9246859-kube-api-access-6589k\" (UniqueName: \"kubernetes.io/projected/ad50a174-edcc-4898-b072-90d6c9246859-kube-api-access-6589k\") pod \"ad50a174-edcc-4898-b072-90d6c9246859\" (UID: \"ad50a174-edcc-4898-b072-90d6c9246859\") " Nov 04 01:34:30.976604 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:30.976519 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad50a174-edcc-4898-b072-90d6c9246859-home\" (UniqueName: \"kubernetes.io/empty-dir/ad50a174-edcc-4898-b072-90d6c9246859-home\") pod \"ad50a174-edcc-4898-b072-90d6c9246859\" (UID: \"ad50a174-edcc-4898-b072-90d6c9246859\") " Nov 04 01:34:30.976604 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:30.976547 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b51f24e-65d2-40fd-8b78-70ee1a593d00-home" pod "1b51f24e-65d2-40fd-8b78-70ee1a593d00" (UID: "1b51f24e-65d2-40fd-8b78-70ee1a593d00"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:30.976604 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:30.976560 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b51f24e-65d2-40fd-8b78-70ee1a593d00-tls-certs\" (UniqueName: \"kubernetes.io/secret/1b51f24e-65d2-40fd-8b78-70ee1a593d00-tls-certs\") pod \"1b51f24e-65d2-40fd-8b78-70ee1a593d00\" (UID: \"1b51f24e-65d2-40fd-8b78-70ee1a593d00\") " Nov 04 01:34:30.977010 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:30.976672 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad50a174-edcc-4898-b072-90d6c9246859-tmp-dir" pod "ad50a174-edcc-4898-b072-90d6c9246859" (UID: "ad50a174-edcc-4898-b072-90d6c9246859"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:30.977010 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:30.977004 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad50a174-edcc-4898-b072-90d6c9246859-home" pod "ad50a174-edcc-4898-b072-90d6c9246859" (UID: "ad50a174-edcc-4898-b072-90d6c9246859"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:30.977342 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:30.977320 7059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad50a174-edcc-4898-b072-90d6c9246859-home\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:30.977418 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:30.977345 7059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad50a174-edcc-4898-b072-90d6c9246859-model-cache\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:30.977418 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:30.977363 7059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b51f24e-65d2-40fd-8b78-70ee1a593d00-model-cache\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:30.977418 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:30.977377 7059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b51f24e-65d2-40fd-8b78-70ee1a593d00-home\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:30.977418 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:30.977391 7059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad50a174-edcc-4898-b072-90d6c9246859-tmp-dir\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:30.977645 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:30.977622 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b51f24e-65d2-40fd-8b78-70ee1a593d00-tmp-dir" pod "1b51f24e-65d2-40fd-8b78-70ee1a593d00" (UID: "1b51f24e-65d2-40fd-8b78-70ee1a593d00"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:30.978721 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:30.978688 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad50a174-edcc-4898-b072-90d6c9246859-tls-certs" pod "ad50a174-edcc-4898-b072-90d6c9246859" (UID: "ad50a174-edcc-4898-b072-90d6c9246859"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:30.978832 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:30.978806 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad50a174-edcc-4898-b072-90d6c9246859-dshm" pod "ad50a174-edcc-4898-b072-90d6c9246859" (UID: "ad50a174-edcc-4898-b072-90d6c9246859"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:30.978832 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:30.978822 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b51f24e-65d2-40fd-8b78-70ee1a593d00-dshm" pod "1b51f24e-65d2-40fd-8b78-70ee1a593d00" (UID: "1b51f24e-65d2-40fd-8b78-70ee1a593d00"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:30.979067 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:30.979048 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b51f24e-65d2-40fd-8b78-70ee1a593d00-tls-certs" pod "1b51f24e-65d2-40fd-8b78-70ee1a593d00" (UID: "1b51f24e-65d2-40fd-8b78-70ee1a593d00"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:30.979436 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:30.979418 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad50a174-edcc-4898-b072-90d6c9246859-kube-api-access-6589k" pod "ad50a174-edcc-4898-b072-90d6c9246859" (UID: "ad50a174-edcc-4898-b072-90d6c9246859"). InnerVolumeSpecName "kube-api-access-6589k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:30.979779 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:30.979760 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b51f24e-65d2-40fd-8b78-70ee1a593d00-kube-api-access-whbzr" pod "1b51f24e-65d2-40fd-8b78-70ee1a593d00" (UID: "1b51f24e-65d2-40fd-8b78-70ee1a593d00"). InnerVolumeSpecName "kube-api-access-whbzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:31.078603 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:31.078523 7059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b51f24e-65d2-40fd-8b78-70ee1a593d00-dshm\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:31.078603 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:31.078552 7059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad50a174-edcc-4898-b072-90d6c9246859-tls-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:31.078603 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:31.078563 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whbzr\" (UniqueName: \"kubernetes.io/projected/1b51f24e-65d2-40fd-8b78-70ee1a593d00-kube-api-access-whbzr\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:31.078603 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:31.078573 7059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b51f24e-65d2-40fd-8b78-70ee1a593d00-tmp-dir\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:31.078603 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:31.078581 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6589k\" (UniqueName: \"kubernetes.io/projected/ad50a174-edcc-4898-b072-90d6c9246859-kube-api-access-6589k\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:31.078603 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:31.078589 7059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b51f24e-65d2-40fd-8b78-70ee1a593d00-tls-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:31.078603 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:31.078597 7059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad50a174-edcc-4898-b072-90d6c9246859-dshm\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:31.200312 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:31.200278 7059 generic.go:356] "Generic (PLEG): container finished" podID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerID="49cb68ad09d1bf01650b6c0f82ecdd96bbd139e9032541b673d1da51560c5a09" exitCode=0 Nov 04 01:34:31.200434 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:31.200323 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" Nov 04 01:34:31.200434 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:31.200356 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" event={"ID":"1b51f24e-65d2-40fd-8b78-70ee1a593d00","Type":"ContainerDied","Data":"49cb68ad09d1bf01650b6c0f82ecdd96bbd139e9032541b673d1da51560c5a09"} Nov 04 01:34:31.200434 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:31.200396 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z" event={"ID":"1b51f24e-65d2-40fd-8b78-70ee1a593d00","Type":"ContainerDied","Data":"9904beb4a3242c05ba8f8e738870c428d549bde1913556f92925d8ce7b7570f4"} Nov 04 01:34:31.200434 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:31.200413 7059 scope.go:122] "RemoveContainer" containerID="49cb68ad09d1bf01650b6c0f82ecdd96bbd139e9032541b673d1da51560c5a09" Nov 04 01:34:31.201981 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:31.201917 7059 generic.go:356] "Generic (PLEG): container finished" podID="ad50a174-edcc-4898-b072-90d6c9246859" containerID="713e989f10ddfc20e3b2695c179d01563a68536411955099c1df730b0e2b265c" exitCode=0 Nov 04 01:34:31.202165 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:31.202021 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" event={"ID":"ad50a174-edcc-4898-b072-90d6c9246859","Type":"ContainerDied","Data":"713e989f10ddfc20e3b2695c179d01563a68536411955099c1df730b0e2b265c"} Nov 04 01:34:31.202165 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:31.202050 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" event={"ID":"ad50a174-edcc-4898-b072-90d6c9246859","Type":"ContainerDied","Data":"ca51bfaf8404d327e5b7814cde58c096dd495e948a2b92db6b0f5233a8e662f9"} Nov 04 01:34:31.202165 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:31.202060 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz" Nov 04 01:34:31.211073 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:31.211053 7059 scope.go:122] "RemoveContainer" containerID="49cb68ad09d1bf01650b6c0f82ecdd96bbd139e9032541b673d1da51560c5a09" Nov 04 01:34:31.211355 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:34:31.211328 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49cb68ad09d1bf01650b6c0f82ecdd96bbd139e9032541b673d1da51560c5a09\": container with ID starting with 49cb68ad09d1bf01650b6c0f82ecdd96bbd139e9032541b673d1da51560c5a09 not found: ID does not exist" containerID="49cb68ad09d1bf01650b6c0f82ecdd96bbd139e9032541b673d1da51560c5a09" Nov 04 01:34:31.211429 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:31.211365 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49cb68ad09d1bf01650b6c0f82ecdd96bbd139e9032541b673d1da51560c5a09"} err="failed to get container status \"49cb68ad09d1bf01650b6c0f82ecdd96bbd139e9032541b673d1da51560c5a09\": rpc error: code = NotFound desc = could not find container \"49cb68ad09d1bf01650b6c0f82ecdd96bbd139e9032541b673d1da51560c5a09\": container with ID starting with 49cb68ad09d1bf01650b6c0f82ecdd96bbd139e9032541b673d1da51560c5a09 not found: ID does not exist" Nov 04 01:34:31.211429 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:31.211387 7059 scope.go:122] "RemoveContainer" containerID="713e989f10ddfc20e3b2695c179d01563a68536411955099c1df730b0e2b265c" Nov 04 01:34:31.246538 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:31.246469 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz"] Nov 04 01:34:31.248966 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:31.248942 7059 scope.go:122] "RemoveContainer" containerID="713e989f10ddfc20e3b2695c179d01563a68536411955099c1df730b0e2b265c" Nov 04 01:34:31.249413 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:34:31.249388 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"713e989f10ddfc20e3b2695c179d01563a68536411955099c1df730b0e2b265c\": container with ID starting with 713e989f10ddfc20e3b2695c179d01563a68536411955099c1df730b0e2b265c not found: ID does not exist" containerID="713e989f10ddfc20e3b2695c179d01563a68536411955099c1df730b0e2b265c" Nov 04 01:34:31.249529 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:31.249422 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"713e989f10ddfc20e3b2695c179d01563a68536411955099c1df730b0e2b265c"} err="failed to get container status \"713e989f10ddfc20e3b2695c179d01563a68536411955099c1df730b0e2b265c\": rpc error: code = NotFound desc = could not find container \"713e989f10ddfc20e3b2695c179d01563a68536411955099c1df730b0e2b265c\": container with ID starting with 713e989f10ddfc20e3b2695c179d01563a68536411955099c1df730b0e2b265c not found: ID does not exist" Nov 04 01:34:31.252904 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:31.252850 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-586541/distinct-w1-586541-kserve-6bd6fc799-59vbz"] Nov 04 01:34:31.259813 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:31.259785 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z"] Nov 04 01:34:31.280595 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:31.280567 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-586541/simulator-w1-586541-kserve-6fc9f9dcb5-52w9z"] Nov 04 01:34:33.179877 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:33.179845 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" path="/var/lib/kubelet/pods/1b51f24e-65d2-40fd-8b78-70ee1a593d00/volumes" Nov 04 01:34:33.180375 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:33.180318 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad50a174-edcc-4898-b072-90d6c9246859" path="/var/lib/kubelet/pods/ad50a174-edcc-4898-b072-90d6c9246859/volumes" Nov 04 01:34:33.675005 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:33.674969 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l"] Nov 04 01:34:33.675512 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:33.675229 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" podUID="f0f8924f-e452-4b2c-8b41-f970764ff963" containerName="istio-proxy" containerID="cri-o://08dfdcd839590cf2ee0e3d7c29edd4feb32fee076c7a6ae91b7745d570b8a82e" gracePeriod=30 Nov 04 01:34:33.687127 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:33.687095 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c"] Nov 04 01:34:33.687368 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:33.687340 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" podUID="f9e034fc-c636-44dd-a58c-b8cf6d3a5734" containerName="istio-proxy" containerID="cri-o://350cc5ac2d6f7c94547d73b5a37f11f40862fa5f593a1504260c1857d142c724" gracePeriod=30 Nov 04 01:34:33.700875 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:33.700849 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44"] Nov 04 01:34:33.701088 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:33.701065 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" podUID="0085890d-7ef5-4947-975c-440894141023" containerName="istio-proxy" containerID="cri-o://d5d64953a1e2923156d28020c73b9da28815ba42cbfa406c02a35d50527fd1fd" gracePeriod=30 Nov 04 01:34:34.343434 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:34.343381 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-9c8bf5-646c6d9774-4862q" podUID="158365b9-7071-4a6d-b2a4-f6b56d795ec1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:34:34.345210 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:34.345181 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9c8bf5-9758b5cf-9hsn2" podUID="142d5751-9524-4f5c-8072-d05d5ca426ad" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:34:34.671353 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:34.671277 7059 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:34:34.671353 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:34.671333 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" podUID="0085890d-7ef5-4947-975c-440894141023" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:34:35.236226 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:35.236192 7059 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:34:35.236396 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:35.236252 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" podUID="f0f8924f-e452-4b2c-8b41-f970764ff963" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:34:36.295964 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:36.295860 7059 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:34:36.295964 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:36.295911 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" podUID="f9e034fc-c636-44dd-a58c-b8cf6d3a5734" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:34:36.483174 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:34:36.483138 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:34:36.642882 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:36.642803 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf"] Nov 04 01:34:36.643075 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:36.643046 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf" podUID="d75e9829-735d-4f1c-b79b-55b46d5cfc01" containerName="maas-api" containerID="cri-o://cee0774be47adcf1df88dc4f33e9431833777e7f9d354c320d04457b8f3bc489" gracePeriod=30 Nov 04 01:34:36.770167 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:36.770051 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt"] Nov 04 01:34:36.773308 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:36.770917 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt" podUID="3eb77296-8f8c-4ed8-8d70-f6cbdb09b888" containerName="maas-api" containerID="cri-o://415a634ba090cdacd3ab4109f467fc070bd88beb17ea444ee4a22210cee09d43" gracePeriod=30 Nov 04 01:34:36.987940 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:36.980138 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf" Nov 04 01:34:37.118434 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.113113 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d75e9829-735d-4f1c-b79b-55b46d5cfc01-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d75e9829-735d-4f1c-b79b-55b46d5cfc01-openshift-service-ca\") pod \"d75e9829-735d-4f1c-b79b-55b46d5cfc01\" (UID: \"d75e9829-735d-4f1c-b79b-55b46d5cfc01\") " Nov 04 01:34:37.118434 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.113257 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d75e9829-735d-4f1c-b79b-55b46d5cfc01-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d75e9829-735d-4f1c-b79b-55b46d5cfc01-maas-api-tls\") pod \"d75e9829-735d-4f1c-b79b-55b46d5cfc01\" (UID: \"d75e9829-735d-4f1c-b79b-55b46d5cfc01\") " Nov 04 01:34:37.118434 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.113310 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d75e9829-735d-4f1c-b79b-55b46d5cfc01-kube-api-access-62pkl\" (UniqueName: \"kubernetes.io/projected/d75e9829-735d-4f1c-b79b-55b46d5cfc01-kube-api-access-62pkl\") pod \"d75e9829-735d-4f1c-b79b-55b46d5cfc01\" (UID: \"d75e9829-735d-4f1c-b79b-55b46d5cfc01\") " Nov 04 01:34:37.118434 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.113354 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d75e9829-735d-4f1c-b79b-55b46d5cfc01-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d75e9829-735d-4f1c-b79b-55b46d5cfc01-metrics-certs\") pod \"d75e9829-735d-4f1c-b79b-55b46d5cfc01\" (UID: \"d75e9829-735d-4f1c-b79b-55b46d5cfc01\") " Nov 04 01:34:37.118434 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.115321 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d75e9829-735d-4f1c-b79b-55b46d5cfc01-openshift-service-ca" pod "d75e9829-735d-4f1c-b79b-55b46d5cfc01" (UID: "d75e9829-735d-4f1c-b79b-55b46d5cfc01"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:37.147834 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.147746 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d75e9829-735d-4f1c-b79b-55b46d5cfc01-maas-api-tls" pod "d75e9829-735d-4f1c-b79b-55b46d5cfc01" (UID: "d75e9829-735d-4f1c-b79b-55b46d5cfc01"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:37.148216 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.148188 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d75e9829-735d-4f1c-b79b-55b46d5cfc01-metrics-certs" pod "d75e9829-735d-4f1c-b79b-55b46d5cfc01" (UID: "d75e9829-735d-4f1c-b79b-55b46d5cfc01"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:37.150582 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.150313 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d75e9829-735d-4f1c-b79b-55b46d5cfc01-kube-api-access-62pkl" pod "d75e9829-735d-4f1c-b79b-55b46d5cfc01" (UID: "d75e9829-735d-4f1c-b79b-55b46d5cfc01"). InnerVolumeSpecName "kube-api-access-62pkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:37.187098 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.187065 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt" Nov 04 01:34:37.215660 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.215635 7059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d75e9829-735d-4f1c-b79b-55b46d5cfc01-openshift-service-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:37.215660 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.215660 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d75e9829-735d-4f1c-b79b-55b46d5cfc01-maas-api-tls\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:37.215828 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.215671 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62pkl\" (UniqueName: \"kubernetes.io/projected/d75e9829-735d-4f1c-b79b-55b46d5cfc01-kube-api-access-62pkl\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:37.215828 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.215680 7059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d75e9829-735d-4f1c-b79b-55b46d5cfc01-metrics-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:37.291997 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.291966 7059 generic.go:356] "Generic (PLEG): container finished" podID="d75e9829-735d-4f1c-b79b-55b46d5cfc01" containerID="cee0774be47adcf1df88dc4f33e9431833777e7f9d354c320d04457b8f3bc489" exitCode=0 Nov 04 01:34:37.292148 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.292019 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf" Nov 04 01:34:37.292148 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.292055 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf" event={"ID":"d75e9829-735d-4f1c-b79b-55b46d5cfc01","Type":"ContainerDied","Data":"cee0774be47adcf1df88dc4f33e9431833777e7f9d354c320d04457b8f3bc489"} Nov 04 01:34:37.292148 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.292117 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf" event={"ID":"d75e9829-735d-4f1c-b79b-55b46d5cfc01","Type":"ContainerDied","Data":"5da7d8541c190fcbd5d0063c887c514f398b264b7b489f979bbc957f42a99129"} Nov 04 01:34:37.292148 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.292142 7059 scope.go:122] "RemoveContainer" containerID="cee0774be47adcf1df88dc4f33e9431833777e7f9d354c320d04457b8f3bc489" Nov 04 01:34:37.294429 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.294402 7059 generic.go:356] "Generic (PLEG): container finished" podID="3eb77296-8f8c-4ed8-8d70-f6cbdb09b888" containerID="415a634ba090cdacd3ab4109f467fc070bd88beb17ea444ee4a22210cee09d43" exitCode=0 Nov 04 01:34:37.294620 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.294450 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt" Nov 04 01:34:37.294620 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.294483 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt" event={"ID":"3eb77296-8f8c-4ed8-8d70-f6cbdb09b888","Type":"ContainerDied","Data":"415a634ba090cdacd3ab4109f467fc070bd88beb17ea444ee4a22210cee09d43"} Nov 04 01:34:37.294620 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.294527 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt" event={"ID":"3eb77296-8f8c-4ed8-8d70-f6cbdb09b888","Type":"ContainerDied","Data":"df117a9947931518924d2d3b7c381998352243c2dbd676c2ca72f4686106d274"} Nov 04 01:34:37.303402 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.303387 7059 scope.go:122] "RemoveContainer" containerID="cee0774be47adcf1df88dc4f33e9431833777e7f9d354c320d04457b8f3bc489" Nov 04 01:34:37.303674 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:34:37.303646 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cee0774be47adcf1df88dc4f33e9431833777e7f9d354c320d04457b8f3bc489\": container with ID starting with cee0774be47adcf1df88dc4f33e9431833777e7f9d354c320d04457b8f3bc489 not found: ID does not exist" containerID="cee0774be47adcf1df88dc4f33e9431833777e7f9d354c320d04457b8f3bc489" Nov 04 01:34:37.303717 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.303670 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cee0774be47adcf1df88dc4f33e9431833777e7f9d354c320d04457b8f3bc489"} err="failed to get container status \"cee0774be47adcf1df88dc4f33e9431833777e7f9d354c320d04457b8f3bc489\": rpc error: code = NotFound desc = could not find container \"cee0774be47adcf1df88dc4f33e9431833777e7f9d354c320d04457b8f3bc489\": container with ID starting with cee0774be47adcf1df88dc4f33e9431833777e7f9d354c320d04457b8f3bc489 not found: ID does not exist" Nov 04 01:34:37.303717 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.303684 7059 scope.go:122] "RemoveContainer" containerID="415a634ba090cdacd3ab4109f467fc070bd88beb17ea444ee4a22210cee09d43" Nov 04 01:34:37.315195 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.315177 7059 scope.go:122] "RemoveContainer" containerID="415a634ba090cdacd3ab4109f467fc070bd88beb17ea444ee4a22210cee09d43" Nov 04 01:34:37.315473 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:34:37.315455 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"415a634ba090cdacd3ab4109f467fc070bd88beb17ea444ee4a22210cee09d43\": container with ID starting with 415a634ba090cdacd3ab4109f467fc070bd88beb17ea444ee4a22210cee09d43 not found: ID does not exist" containerID="415a634ba090cdacd3ab4109f467fc070bd88beb17ea444ee4a22210cee09d43" Nov 04 01:34:37.315536 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.315484 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"415a634ba090cdacd3ab4109f467fc070bd88beb17ea444ee4a22210cee09d43"} err="failed to get container status \"415a634ba090cdacd3ab4109f467fc070bd88beb17ea444ee4a22210cee09d43\": rpc error: code = NotFound desc = could not find container \"415a634ba090cdacd3ab4109f467fc070bd88beb17ea444ee4a22210cee09d43\": container with ID starting with 415a634ba090cdacd3ab4109f467fc070bd88beb17ea444ee4a22210cee09d43 not found: ID does not exist" Nov 04 01:34:37.316832 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.316815 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eb77296-8f8c-4ed8-8d70-f6cbdb09b888-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3eb77296-8f8c-4ed8-8d70-f6cbdb09b888-maas-api-tls\") pod \"3eb77296-8f8c-4ed8-8d70-f6cbdb09b888\" (UID: \"3eb77296-8f8c-4ed8-8d70-f6cbdb09b888\") " Nov 04 01:34:37.316972 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.316955 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eb77296-8f8c-4ed8-8d70-f6cbdb09b888-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb77296-8f8c-4ed8-8d70-f6cbdb09b888-openshift-service-ca\") pod \"3eb77296-8f8c-4ed8-8d70-f6cbdb09b888\" (UID: \"3eb77296-8f8c-4ed8-8d70-f6cbdb09b888\") " Nov 04 01:34:37.317129 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.317111 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eb77296-8f8c-4ed8-8d70-f6cbdb09b888-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eb77296-8f8c-4ed8-8d70-f6cbdb09b888-metrics-certs\") pod \"3eb77296-8f8c-4ed8-8d70-f6cbdb09b888\" (UID: \"3eb77296-8f8c-4ed8-8d70-f6cbdb09b888\") " Nov 04 01:34:37.317224 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.317207 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb77296-8f8c-4ed8-8d70-f6cbdb09b888-kube-api-access-7c6xt\" (UniqueName: \"kubernetes.io/projected/3eb77296-8f8c-4ed8-8d70-f6cbdb09b888-kube-api-access-7c6xt\") pod \"3eb77296-8f8c-4ed8-8d70-f6cbdb09b888\" (UID: \"3eb77296-8f8c-4ed8-8d70-f6cbdb09b888\") " Nov 04 01:34:37.317329 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.317310 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb77296-8f8c-4ed8-8d70-f6cbdb09b888-openshift-service-ca" pod "3eb77296-8f8c-4ed8-8d70-f6cbdb09b888" (UID: "3eb77296-8f8c-4ed8-8d70-f6cbdb09b888"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:37.317784 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.317761 7059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb77296-8f8c-4ed8-8d70-f6cbdb09b888-openshift-service-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:37.319251 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.319228 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb77296-8f8c-4ed8-8d70-f6cbdb09b888-metrics-certs" pod "3eb77296-8f8c-4ed8-8d70-f6cbdb09b888" (UID: "3eb77296-8f8c-4ed8-8d70-f6cbdb09b888"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:37.319506 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.319457 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb77296-8f8c-4ed8-8d70-f6cbdb09b888-maas-api-tls" pod "3eb77296-8f8c-4ed8-8d70-f6cbdb09b888" (UID: "3eb77296-8f8c-4ed8-8d70-f6cbdb09b888"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:37.324039 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.323993 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb77296-8f8c-4ed8-8d70-f6cbdb09b888-kube-api-access-7c6xt" pod "3eb77296-8f8c-4ed8-8d70-f6cbdb09b888" (UID: "3eb77296-8f8c-4ed8-8d70-f6cbdb09b888"). InnerVolumeSpecName "kube-api-access-7c6xt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:37.330321 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.329025 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf"] Nov 04 01:34:37.343049 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.343009 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-586541-6c866f9865-wzwgf"] Nov 04 01:34:37.418879 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.418800 7059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eb77296-8f8c-4ed8-8d70-f6cbdb09b888-metrics-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:37.418879 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.418831 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c6xt\" (UniqueName: \"kubernetes.io/projected/3eb77296-8f8c-4ed8-8d70-f6cbdb09b888-kube-api-access-7c6xt\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:37.418879 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.418845 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3eb77296-8f8c-4ed8-8d70-f6cbdb09b888-maas-api-tls\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:37.628832 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.628779 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt"] Nov 04 01:34:37.633955 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.633909 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-98f6f6-5954b65b89-gnklt"] Nov 04 01:34:37.928190 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:37.928164 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-9c8bf5-646c6d9774-4862q" Nov 04 01:34:38.034064 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:38.032608 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/158365b9-7071-4a6d-b2a4-f6b56d795ec1-kube-api-access-r25h8\" (UniqueName: \"kubernetes.io/projected/158365b9-7071-4a6d-b2a4-f6b56d795ec1-kube-api-access-r25h8\") pod \"158365b9-7071-4a6d-b2a4-f6b56d795ec1\" (UID: \"158365b9-7071-4a6d-b2a4-f6b56d795ec1\") " Nov 04 01:34:38.034064 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:38.032698 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/158365b9-7071-4a6d-b2a4-f6b56d795ec1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/158365b9-7071-4a6d-b2a4-f6b56d795ec1-plugins-config-volume\") pod \"158365b9-7071-4a6d-b2a4-f6b56d795ec1\" (UID: \"158365b9-7071-4a6d-b2a4-f6b56d795ec1\") " Nov 04 01:34:38.034064 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:38.033812 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/158365b9-7071-4a6d-b2a4-f6b56d795ec1-plugins-config-volume" pod "158365b9-7071-4a6d-b2a4-f6b56d795ec1" (UID: "158365b9-7071-4a6d-b2a4-f6b56d795ec1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:38.045459 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:38.045396 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/158365b9-7071-4a6d-b2a4-f6b56d795ec1-kube-api-access-r25h8" pod "158365b9-7071-4a6d-b2a4-f6b56d795ec1" (UID: "158365b9-7071-4a6d-b2a4-f6b56d795ec1"). InnerVolumeSpecName "kube-api-access-r25h8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:38.085105 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:38.085080 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9c8bf5-9758b5cf-9hsn2" Nov 04 01:34:38.135263 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:38.135221 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r25h8\" (UniqueName: \"kubernetes.io/projected/158365b9-7071-4a6d-b2a4-f6b56d795ec1-kube-api-access-r25h8\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:38.135263 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:38.135262 7059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/158365b9-7071-4a6d-b2a4-f6b56d795ec1-plugins-config-volume\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:38.242177 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:38.236658 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/142d5751-9524-4f5c-8072-d05d5ca426ad-kube-api-access-6k4c2\" (UniqueName: \"kubernetes.io/projected/142d5751-9524-4f5c-8072-d05d5ca426ad-kube-api-access-6k4c2\") pod \"142d5751-9524-4f5c-8072-d05d5ca426ad\" (UID: \"142d5751-9524-4f5c-8072-d05d5ca426ad\") " Nov 04 01:34:38.257950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:38.248628 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/142d5751-9524-4f5c-8072-d05d5ca426ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/142d5751-9524-4f5c-8072-d05d5ca426ad-plugins-config-volume\") pod \"142d5751-9524-4f5c-8072-d05d5ca426ad\" (UID: \"142d5751-9524-4f5c-8072-d05d5ca426ad\") " Nov 04 01:34:38.264217 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:38.264154 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/142d5751-9524-4f5c-8072-d05d5ca426ad-plugins-config-volume" pod "142d5751-9524-4f5c-8072-d05d5ca426ad" (UID: "142d5751-9524-4f5c-8072-d05d5ca426ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:38.287023 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:38.286893 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/142d5751-9524-4f5c-8072-d05d5ca426ad-kube-api-access-6k4c2" pod "142d5751-9524-4f5c-8072-d05d5ca426ad" (UID: "142d5751-9524-4f5c-8072-d05d5ca426ad"). InnerVolumeSpecName "kube-api-access-6k4c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:38.319468 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:38.319427 7059 generic.go:356] "Generic (PLEG): container finished" podID="142d5751-9524-4f5c-8072-d05d5ca426ad" containerID="cbc20e0c0e789a803cca3793bf9f27ec3d0cc9841d566664fcff950c302a7a52" exitCode=0 Nov 04 01:34:38.320028 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:38.319490 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9c8bf5-9758b5cf-9hsn2" Nov 04 01:34:38.320028 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:38.319545 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9c8bf5-9758b5cf-9hsn2" event={"ID":"142d5751-9524-4f5c-8072-d05d5ca426ad","Type":"ContainerDied","Data":"cbc20e0c0e789a803cca3793bf9f27ec3d0cc9841d566664fcff950c302a7a52"} Nov 04 01:34:38.320028 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:38.319590 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-9c8bf5-9758b5cf-9hsn2" event={"ID":"142d5751-9524-4f5c-8072-d05d5ca426ad","Type":"ContainerDied","Data":"a7c54f68c6097a30b5ec38cfa46ec9b784feff54cd472689024e20db32679679"} Nov 04 01:34:38.320028 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:38.319614 7059 scope.go:122] "RemoveContainer" containerID="cbc20e0c0e789a803cca3793bf9f27ec3d0cc9841d566664fcff950c302a7a52" Nov 04 01:34:38.322757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:38.322726 7059 generic.go:356] "Generic (PLEG): container finished" podID="158365b9-7071-4a6d-b2a4-f6b56d795ec1" containerID="635f93a606a4c10f2fe4d9f15a86fa07dcd9fabfeb5b131212eac77cfc4095d9" exitCode=0 Nov 04 01:34:38.322913 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:38.322898 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-9c8bf5-646c6d9774-4862q" Nov 04 01:34:38.323110 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:38.323082 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-9c8bf5-646c6d9774-4862q" event={"ID":"158365b9-7071-4a6d-b2a4-f6b56d795ec1","Type":"ContainerDied","Data":"635f93a606a4c10f2fe4d9f15a86fa07dcd9fabfeb5b131212eac77cfc4095d9"} Nov 04 01:34:38.323873 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:38.323254 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-9c8bf5-646c6d9774-4862q" event={"ID":"158365b9-7071-4a6d-b2a4-f6b56d795ec1","Type":"ContainerDied","Data":"35626fdc097ee9ee026a4717d7ad078fef2dda92283c808b77d371bddac1c330"} Nov 04 01:34:38.363546 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:38.363506 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k4c2\" (UniqueName: \"kubernetes.io/projected/142d5751-9524-4f5c-8072-d05d5ca426ad-kube-api-access-6k4c2\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:38.363546 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:38.363550 7059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/142d5751-9524-4f5c-8072-d05d5ca426ad-plugins-config-volume\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:38.365461 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:38.365436 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-9c8bf5-9758b5cf-9hsn2"] Nov 04 01:34:38.368776 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:38.368316 7059 scope.go:122] "RemoveContainer" containerID="cbc20e0c0e789a803cca3793bf9f27ec3d0cc9841d566664fcff950c302a7a52" Nov 04 01:34:38.368866 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:34:38.368840 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbc20e0c0e789a803cca3793bf9f27ec3d0cc9841d566664fcff950c302a7a52\": container with ID starting with cbc20e0c0e789a803cca3793bf9f27ec3d0cc9841d566664fcff950c302a7a52 not found: ID does not exist" containerID="cbc20e0c0e789a803cca3793bf9f27ec3d0cc9841d566664fcff950c302a7a52" Nov 04 01:34:38.368945 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:38.368879 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbc20e0c0e789a803cca3793bf9f27ec3d0cc9841d566664fcff950c302a7a52"} err="failed to get container status \"cbc20e0c0e789a803cca3793bf9f27ec3d0cc9841d566664fcff950c302a7a52\": rpc error: code = NotFound desc = could not find container \"cbc20e0c0e789a803cca3793bf9f27ec3d0cc9841d566664fcff950c302a7a52\": container with ID starting with cbc20e0c0e789a803cca3793bf9f27ec3d0cc9841d566664fcff950c302a7a52 not found: ID does not exist" Nov 04 01:34:38.368945 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:38.368906 7059 scope.go:122] "RemoveContainer" containerID="635f93a606a4c10f2fe4d9f15a86fa07dcd9fabfeb5b131212eac77cfc4095d9" Nov 04 01:34:38.370180 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:38.370116 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-9c8bf5-9758b5cf-9hsn2"] Nov 04 01:34:38.380232 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:38.380214 7059 scope.go:122] "RemoveContainer" containerID="635f93a606a4c10f2fe4d9f15a86fa07dcd9fabfeb5b131212eac77cfc4095d9" Nov 04 01:34:38.380552 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:34:38.380520 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"635f93a606a4c10f2fe4d9f15a86fa07dcd9fabfeb5b131212eac77cfc4095d9\": container with ID starting with 635f93a606a4c10f2fe4d9f15a86fa07dcd9fabfeb5b131212eac77cfc4095d9 not found: ID does not exist" containerID="635f93a606a4c10f2fe4d9f15a86fa07dcd9fabfeb5b131212eac77cfc4095d9" Nov 04 01:34:38.380648 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:38.380563 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"635f93a606a4c10f2fe4d9f15a86fa07dcd9fabfeb5b131212eac77cfc4095d9"} err="failed to get container status \"635f93a606a4c10f2fe4d9f15a86fa07dcd9fabfeb5b131212eac77cfc4095d9\": rpc error: code = NotFound desc = could not find container \"635f93a606a4c10f2fe4d9f15a86fa07dcd9fabfeb5b131212eac77cfc4095d9\": container with ID starting with 635f93a606a4c10f2fe4d9f15a86fa07dcd9fabfeb5b131212eac77cfc4095d9 not found: ID does not exist" Nov 04 01:34:38.382217 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:38.382197 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-9c8bf5-646c6d9774-4862q"] Nov 04 01:34:38.388378 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:38.388356 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-9c8bf5-646c6d9774-4862q"] Nov 04 01:34:39.015599 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.015565 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:34:39.055848 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.055538 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:34:39.061115 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.061092 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:34:39.178822 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.178740 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="142d5751-9524-4f5c-8072-d05d5ca426ad" path="/var/lib/kubelet/pods/142d5751-9524-4f5c-8072-d05d5ca426ad/volumes" Nov 04 01:34:39.179245 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.179229 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="158365b9-7071-4a6d-b2a4-f6b56d795ec1" path="/var/lib/kubelet/pods/158365b9-7071-4a6d-b2a4-f6b56d795ec1/volumes" Nov 04 01:34:39.179625 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.179613 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eb77296-8f8c-4ed8-8d70-f6cbdb09b888" path="/var/lib/kubelet/pods/3eb77296-8f8c-4ed8-8d70-f6cbdb09b888/volumes" Nov 04 01:34:39.180427 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.180409 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d75e9829-735d-4f1c-b79b-55b46d5cfc01" path="/var/lib/kubelet/pods/d75e9829-735d-4f1c-b79b-55b46d5cfc01/volumes" Nov 04 01:34:39.186962 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.186939 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istio-token\" (UniqueName: \"kubernetes.io/projected/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istio-token\") pod \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " Nov 04 01:34:39.187075 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.187007 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-workload-certs\") pod \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " Nov 04 01:34:39.187075 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.187031 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-istio-data\") pod \"f0f8924f-e452-4b2c-8b41-f970764ff963\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " Nov 04 01:34:39.187075 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.187058 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0f8924f-e452-4b2c-8b41-f970764ff963-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f8924f-e452-4b2c-8b41-f970764ff963-istio-podinfo\") pod \"f0f8924f-e452-4b2c-8b41-f970764ff963\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " Nov 04 01:34:39.187226 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.187103 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istio-envoy\") pod \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " Nov 04 01:34:39.187226 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.187135 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-kube-api-access-dkqjw\" (UniqueName: \"kubernetes.io/projected/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-kube-api-access-dkqjw\") pod \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " Nov 04 01:34:39.187226 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.187164 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0085890d-7ef5-4947-975c-440894141023-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0085890d-7ef5-4947-975c-440894141023-istio-podinfo\") pod \"0085890d-7ef5-4947-975c-440894141023\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " Nov 04 01:34:39.187371 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.187324 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-workload-certs" pod "f9e034fc-c636-44dd-a58c-b8cf6d3a5734" (UID: "f9e034fc-c636-44dd-a58c-b8cf6d3a5734"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.187371 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.187338 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-istio-data" pod "f0f8924f-e452-4b2c-8b41-f970764ff963" (UID: "f0f8924f-e452-4b2c-8b41-f970764ff963"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.187573 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.187552 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istiod-ca-cert\") pod \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " Nov 04 01:34:39.188000 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.187979 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-workload-certs\") pod \"f0f8924f-e452-4b2c-8b41-f970764ff963\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " Nov 04 01:34:39.188233 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.188214 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-workload-certs\") pod \"0085890d-7ef5-4947-975c-440894141023\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " Nov 04 01:34:39.188432 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.188306 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-workload-certs" pod "f0f8924f-e452-4b2c-8b41-f970764ff963" (UID: "f0f8924f-e452-4b2c-8b41-f970764ff963"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.188531 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.188510 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-workload-certs" pod "0085890d-7ef5-4947-975c-440894141023" (UID: "0085890d-7ef5-4947-975c-440894141023"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.188878 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.188684 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istiod-ca-cert" pod "f9e034fc-c636-44dd-a58c-b8cf6d3a5734" (UID: "f9e034fc-c636-44dd-a58c-b8cf6d3a5734"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:39.189011 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.188990 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0085890d-7ef5-4947-975c-440894141023-istio-token\" (UniqueName: \"kubernetes.io/projected/0085890d-7ef5-4947-975c-440894141023-istio-token\") pod \"0085890d-7ef5-4947-975c-440894141023\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " Nov 04 01:34:39.189245 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.189227 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-credential-socket\") pod \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " Nov 04 01:34:39.189403 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.189388 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istio-podinfo\") pod \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " Nov 04 01:34:39.189608 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.189583 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-workload-socket\") pod \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " Nov 04 01:34:39.189608 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.189599 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-credential-socket" pod "f9e034fc-c636-44dd-a58c-b8cf6d3a5734" (UID: "f9e034fc-c636-44dd-a58c-b8cf6d3a5734"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.189754 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.189666 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0f8924f-e452-4b2c-8b41-f970764ff963-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f8924f-e452-4b2c-8b41-f970764ff963-istiod-ca-cert\") pod \"f0f8924f-e452-4b2c-8b41-f970764ff963\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " Nov 04 01:34:39.189754 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.189717 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-istio-envoy\") pod \"f0f8924f-e452-4b2c-8b41-f970764ff963\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " Nov 04 01:34:39.189911 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.189751 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istio-data\") pod \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\" (UID: \"f9e034fc-c636-44dd-a58c-b8cf6d3a5734\") " Nov 04 01:34:39.189911 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.189810 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-workload-socket\") pod \"f0f8924f-e452-4b2c-8b41-f970764ff963\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " Nov 04 01:34:39.189911 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.189849 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-kube-api-access-dkqjw" pod "f9e034fc-c636-44dd-a58c-b8cf6d3a5734" (UID: "f9e034fc-c636-44dd-a58c-b8cf6d3a5734"). InnerVolumeSpecName "kube-api-access-dkqjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:39.189911 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.189874 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-workload-socket\") pod \"0085890d-7ef5-4947-975c-440894141023\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " Nov 04 01:34:39.189911 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.189870 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-workload-socket" pod "f9e034fc-c636-44dd-a58c-b8cf6d3a5734" (UID: "f9e034fc-c636-44dd-a58c-b8cf6d3a5734"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.189911 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.189917 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-credential-socket\") pod \"f0f8924f-e452-4b2c-8b41-f970764ff963\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " Nov 04 01:34:39.190259 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.190006 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0085890d-7ef5-4947-975c-440894141023-kube-api-access-qj6sr\" (UniqueName: \"kubernetes.io/projected/0085890d-7ef5-4947-975c-440894141023-kube-api-access-qj6sr\") pod \"0085890d-7ef5-4947-975c-440894141023\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " Nov 04 01:34:39.190259 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.190065 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0f8924f-e452-4b2c-8b41-f970764ff963-istiod-ca-cert" pod "f0f8924f-e452-4b2c-8b41-f970764ff963" (UID: "f0f8924f-e452-4b2c-8b41-f970764ff963"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:39.190259 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.190149 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-workload-socket" pod "f0f8924f-e452-4b2c-8b41-f970764ff963" (UID: "f0f8924f-e452-4b2c-8b41-f970764ff963"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.190259 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.190075 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0f8924f-e452-4b2c-8b41-f970764ff963-kube-api-access-c6hcp\" (UniqueName: \"kubernetes.io/projected/f0f8924f-e452-4b2c-8b41-f970764ff963-kube-api-access-c6hcp\") pod \"f0f8924f-e452-4b2c-8b41-f970764ff963\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " Nov 04 01:34:39.190259 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.190134 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istio-data" pod "f9e034fc-c636-44dd-a58c-b8cf6d3a5734" (UID: "f9e034fc-c636-44dd-a58c-b8cf6d3a5734"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.190259 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.190207 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-istio-data\") pod \"0085890d-7ef5-4947-975c-440894141023\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " Nov 04 01:34:39.190259 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.190240 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0085890d-7ef5-4947-975c-440894141023-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0085890d-7ef5-4947-975c-440894141023-istiod-ca-cert\") pod \"0085890d-7ef5-4947-975c-440894141023\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " Nov 04 01:34:39.190611 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.190282 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-istio-envoy\") pod \"0085890d-7ef5-4947-975c-440894141023\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " Nov 04 01:34:39.190611 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.190366 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0f8924f-e452-4b2c-8b41-f970764ff963-istio-token\" (UniqueName: \"kubernetes.io/projected/f0f8924f-e452-4b2c-8b41-f970764ff963-istio-token\") pod \"f0f8924f-e452-4b2c-8b41-f970764ff963\" (UID: \"f0f8924f-e452-4b2c-8b41-f970764ff963\") " Nov 04 01:34:39.190611 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.190400 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-credential-socket\") pod \"0085890d-7ef5-4947-975c-440894141023\" (UID: \"0085890d-7ef5-4947-975c-440894141023\") " Nov 04 01:34:39.190611 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.190502 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-istio-data" pod "0085890d-7ef5-4947-975c-440894141023" (UID: "0085890d-7ef5-4947-975c-440894141023"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.190611 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.190547 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-workload-socket" pod "0085890d-7ef5-4947-975c-440894141023" (UID: "0085890d-7ef5-4947-975c-440894141023"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.190611 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.190583 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-credential-socket" pod "f0f8924f-e452-4b2c-8b41-f970764ff963" (UID: "f0f8924f-e452-4b2c-8b41-f970764ff963"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.190973 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.190610 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0085890d-7ef5-4947-975c-440894141023-istiod-ca-cert" pod "0085890d-7ef5-4947-975c-440894141023" (UID: "0085890d-7ef5-4947-975c-440894141023"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:39.190973 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.190784 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istio-envoy" pod "f9e034fc-c636-44dd-a58c-b8cf6d3a5734" (UID: "f9e034fc-c636-44dd-a58c-b8cf6d3a5734"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.191082 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.191050 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-credential-socket" pod "0085890d-7ef5-4947-975c-440894141023" (UID: "0085890d-7ef5-4947-975c-440894141023"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.191228 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.191142 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.191228 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.191166 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.191228 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.191179 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.191228 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.191190 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0f8924f-e452-4b2c-8b41-f970764ff963-istio-podinfo" pod "f0f8924f-e452-4b2c-8b41-f970764ff963" (UID: "f0f8924f-e452-4b2c-8b41-f970764ff963"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:34:39.191228 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.191195 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkqjw\" (UniqueName: \"kubernetes.io/projected/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-kube-api-access-dkqjw\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.191513 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.191235 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.191513 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.191261 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.191513 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.191274 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.191513 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.191290 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.191513 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.191303 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.191513 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.191317 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f8924f-e452-4b2c-8b41-f970764ff963-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.191513 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.191332 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.191513 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.191345 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.191513 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.191357 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.191513 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.191371 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.191513 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.191385 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.191513 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.191400 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0085890d-7ef5-4947-975c-440894141023-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.191513 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.191508 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istio-token" pod "f9e034fc-c636-44dd-a58c-b8cf6d3a5734" (UID: "f9e034fc-c636-44dd-a58c-b8cf6d3a5734"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:39.192027 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.192000 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0085890d-7ef5-4947-975c-440894141023-istio-podinfo" pod "0085890d-7ef5-4947-975c-440894141023" (UID: "0085890d-7ef5-4947-975c-440894141023"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:34:39.192435 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.192409 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istio-podinfo" pod "f9e034fc-c636-44dd-a58c-b8cf6d3a5734" (UID: "f9e034fc-c636-44dd-a58c-b8cf6d3a5734"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:34:39.193005 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.192981 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0085890d-7ef5-4947-975c-440894141023-istio-token" pod "0085890d-7ef5-4947-975c-440894141023" (UID: "0085890d-7ef5-4947-975c-440894141023"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:39.193227 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.193195 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0085890d-7ef5-4947-975c-440894141023-kube-api-access-qj6sr" pod "0085890d-7ef5-4947-975c-440894141023" (UID: "0085890d-7ef5-4947-975c-440894141023"). InnerVolumeSpecName "kube-api-access-qj6sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:39.193317 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.193222 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f8924f-e452-4b2c-8b41-f970764ff963-istio-token" pod "f0f8924f-e452-4b2c-8b41-f970764ff963" (UID: "f0f8924f-e452-4b2c-8b41-f970764ff963"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:39.193317 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.193260 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f8924f-e452-4b2c-8b41-f970764ff963-kube-api-access-c6hcp" pod "f0f8924f-e452-4b2c-8b41-f970764ff963" (UID: "f0f8924f-e452-4b2c-8b41-f970764ff963"). InnerVolumeSpecName "kube-api-access-c6hcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:39.193738 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.193718 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-istio-envoy" pod "f0f8924f-e452-4b2c-8b41-f970764ff963" (UID: "f0f8924f-e452-4b2c-8b41-f970764ff963"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.194260 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.194242 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-istio-envoy" pod "0085890d-7ef5-4947-975c-440894141023" (UID: "0085890d-7ef5-4947-975c-440894141023"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.292733 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.292690 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj6sr\" (UniqueName: \"kubernetes.io/projected/0085890d-7ef5-4947-975c-440894141023-kube-api-access-qj6sr\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.292733 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.292724 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6hcp\" (UniqueName: \"kubernetes.io/projected/f0f8924f-e452-4b2c-8b41-f970764ff963-kube-api-access-c6hcp\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.292733 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.292739 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.293013 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.292754 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0f8924f-e452-4b2c-8b41-f970764ff963-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.293013 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.292766 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0085890d-7ef5-4947-975c-440894141023-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.293013 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.292778 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.293013 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.292789 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f8924f-e452-4b2c-8b41-f970764ff963-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.293013 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.292803 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0085890d-7ef5-4947-975c-440894141023-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.293013 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.292814 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0085890d-7ef5-4947-975c-440894141023-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.293013 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.292825 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9e034fc-c636-44dd-a58c-b8cf6d3a5734-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.293013 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.292840 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f8924f-e452-4b2c-8b41-f970764ff963-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.342316 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.342282 7059 generic.go:356] "Generic (PLEG): container finished" podID="f0f8924f-e452-4b2c-8b41-f970764ff963" containerID="08dfdcd839590cf2ee0e3d7c29edd4feb32fee076c7a6ae91b7745d570b8a82e" exitCode=0 Nov 04 01:34:39.342761 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.342345 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" Nov 04 01:34:39.342761 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.342371 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" event={"ID":"f0f8924f-e452-4b2c-8b41-f970764ff963","Type":"ContainerDied","Data":"08dfdcd839590cf2ee0e3d7c29edd4feb32fee076c7a6ae91b7745d570b8a82e"} Nov 04 01:34:39.342761 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.342424 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l" event={"ID":"f0f8924f-e452-4b2c-8b41-f970764ff963","Type":"ContainerDied","Data":"25c24397771825b4d0c117b6f1ce67f36bf4e80b480c6bed9dd39f7ad4b5c74e"} Nov 04 01:34:39.342761 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.342448 7059 scope.go:122] "RemoveContainer" containerID="08dfdcd839590cf2ee0e3d7c29edd4feb32fee076c7a6ae91b7745d570b8a82e" Nov 04 01:34:39.344230 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.344207 7059 generic.go:356] "Generic (PLEG): container finished" podID="f9e034fc-c636-44dd-a58c-b8cf6d3a5734" containerID="350cc5ac2d6f7c94547d73b5a37f11f40862fa5f593a1504260c1857d142c724" exitCode=0 Nov 04 01:34:39.344377 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.344243 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" event={"ID":"f9e034fc-c636-44dd-a58c-b8cf6d3a5734","Type":"ContainerDied","Data":"350cc5ac2d6f7c94547d73b5a37f11f40862fa5f593a1504260c1857d142c724"} Nov 04 01:34:39.344377 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.344290 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" event={"ID":"f9e034fc-c636-44dd-a58c-b8cf6d3a5734","Type":"ContainerDied","Data":"d4c3e407432dc4ad7309aba478b6f734900d93ea775a2d89de9fc3ec262ba90e"} Nov 04 01:34:39.344377 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.344290 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c" Nov 04 01:34:39.345806 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.345779 7059 generic.go:356] "Generic (PLEG): container finished" podID="0085890d-7ef5-4947-975c-440894141023" containerID="d5d64953a1e2923156d28020c73b9da28815ba42cbfa406c02a35d50527fd1fd" exitCode=0 Nov 04 01:34:39.345915 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.345827 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" Nov 04 01:34:39.345915 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.345845 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" event={"ID":"0085890d-7ef5-4947-975c-440894141023","Type":"ContainerDied","Data":"d5d64953a1e2923156d28020c73b9da28815ba42cbfa406c02a35d50527fd1fd"} Nov 04 01:34:39.345915 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.345882 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44" event={"ID":"0085890d-7ef5-4947-975c-440894141023","Type":"ContainerDied","Data":"b53e837e5475edaec76d7d5ed1902f632222bd9f9e239fc3bd2e3d8e773b42db"} Nov 04 01:34:39.356453 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.356436 7059 scope.go:122] "RemoveContainer" containerID="08dfdcd839590cf2ee0e3d7c29edd4feb32fee076c7a6ae91b7745d570b8a82e" Nov 04 01:34:39.356761 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:34:39.356734 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08dfdcd839590cf2ee0e3d7c29edd4feb32fee076c7a6ae91b7745d570b8a82e\": container with ID starting with 08dfdcd839590cf2ee0e3d7c29edd4feb32fee076c7a6ae91b7745d570b8a82e not found: ID does not exist" containerID="08dfdcd839590cf2ee0e3d7c29edd4feb32fee076c7a6ae91b7745d570b8a82e" Nov 04 01:34:39.356879 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.356771 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08dfdcd839590cf2ee0e3d7c29edd4feb32fee076c7a6ae91b7745d570b8a82e"} err="failed to get container status \"08dfdcd839590cf2ee0e3d7c29edd4feb32fee076c7a6ae91b7745d570b8a82e\": rpc error: code = NotFound desc = could not find container \"08dfdcd839590cf2ee0e3d7c29edd4feb32fee076c7a6ae91b7745d570b8a82e\": container with ID starting with 08dfdcd839590cf2ee0e3d7c29edd4feb32fee076c7a6ae91b7745d570b8a82e not found: ID does not exist" Nov 04 01:34:39.356879 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.356790 7059 scope.go:122] "RemoveContainer" containerID="350cc5ac2d6f7c94547d73b5a37f11f40862fa5f593a1504260c1857d142c724" Nov 04 01:34:39.367288 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.367269 7059 scope.go:122] "RemoveContainer" containerID="350cc5ac2d6f7c94547d73b5a37f11f40862fa5f593a1504260c1857d142c724" Nov 04 01:34:39.367555 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:34:39.367536 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"350cc5ac2d6f7c94547d73b5a37f11f40862fa5f593a1504260c1857d142c724\": container with ID starting with 350cc5ac2d6f7c94547d73b5a37f11f40862fa5f593a1504260c1857d142c724 not found: ID does not exist" containerID="350cc5ac2d6f7c94547d73b5a37f11f40862fa5f593a1504260c1857d142c724" Nov 04 01:34:39.367604 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.367562 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"350cc5ac2d6f7c94547d73b5a37f11f40862fa5f593a1504260c1857d142c724"} err="failed to get container status \"350cc5ac2d6f7c94547d73b5a37f11f40862fa5f593a1504260c1857d142c724\": rpc error: code = NotFound desc = could not find container \"350cc5ac2d6f7c94547d73b5a37f11f40862fa5f593a1504260c1857d142c724\": container with ID starting with 350cc5ac2d6f7c94547d73b5a37f11f40862fa5f593a1504260c1857d142c724 not found: ID does not exist" Nov 04 01:34:39.367604 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.367578 7059 scope.go:122] "RemoveContainer" containerID="d5d64953a1e2923156d28020c73b9da28815ba42cbfa406c02a35d50527fd1fd" Nov 04 01:34:39.379664 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.379650 7059 scope.go:122] "RemoveContainer" containerID="d5d64953a1e2923156d28020c73b9da28815ba42cbfa406c02a35d50527fd1fd" Nov 04 01:34:39.380067 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:34:39.380041 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5d64953a1e2923156d28020c73b9da28815ba42cbfa406c02a35d50527fd1fd\": container with ID starting with d5d64953a1e2923156d28020c73b9da28815ba42cbfa406c02a35d50527fd1fd not found: ID does not exist" containerID="d5d64953a1e2923156d28020c73b9da28815ba42cbfa406c02a35d50527fd1fd" Nov 04 01:34:39.380161 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.380072 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5d64953a1e2923156d28020c73b9da28815ba42cbfa406c02a35d50527fd1fd"} err="failed to get container status \"d5d64953a1e2923156d28020c73b9da28815ba42cbfa406c02a35d50527fd1fd\": rpc error: code = NotFound desc = could not find container \"d5d64953a1e2923156d28020c73b9da28815ba42cbfa406c02a35d50527fd1fd\": container with ID starting with d5d64953a1e2923156d28020c73b9da28815ba42cbfa406c02a35d50527fd1fd not found: ID does not exist" Nov 04 01:34:39.393559 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.393530 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c"] Nov 04 01:34:39.409008 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.408975 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-vx47c"] Nov 04 01:34:39.428881 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.428827 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44"] Nov 04 01:34:39.432069 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.432050 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hqn44"] Nov 04 01:34:39.447756 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.447733 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l"] Nov 04 01:34:39.453948 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:39.453912 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-l9k7l"] Nov 04 01:34:41.178637 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:41.178599 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0085890d-7ef5-4947-975c-440894141023" path="/var/lib/kubelet/pods/0085890d-7ef5-4947-975c-440894141023/volumes" Nov 04 01:34:41.179204 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:41.179186 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0f8924f-e452-4b2c-8b41-f970764ff963" path="/var/lib/kubelet/pods/f0f8924f-e452-4b2c-8b41-f970764ff963/volumes" Nov 04 01:34:41.179613 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:41.179599 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9e034fc-c636-44dd-a58c-b8cf6d3a5734" path="/var/lib/kubelet/pods/f9e034fc-c636-44dd-a58c-b8cf6d3a5734/volumes" Nov 04 01:34:44.300904 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:44.300867 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn"] Nov 04 01:34:44.301420 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:44.301149 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" podUID="7e4a1625-eab6-4e98-b8e8-0ccc12edc542" containerName="istio-proxy" containerID="cri-o://f2681d5814ddc28822bd4a6ca61d61351bcc6b7f07b3632583ad4fbc9de41525" gracePeriod=30 Nov 04 01:34:49.288760 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.288718 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2"] Nov 04 01:34:49.289386 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.289355 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2" podUID="4630234e-0999-43f6-9221-007ed3f80001" containerName="maas-api" containerID="cri-o://7f9ad845d074fe5ec22555a7939d0416b5b317108fdfec0663979b3321a8227b" gracePeriod=30 Nov 04 01:34:49.429970 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.427440 7059 generic.go:356] "Generic (PLEG): container finished" podID="4630234e-0999-43f6-9221-007ed3f80001" containerID="7f9ad845d074fe5ec22555a7939d0416b5b317108fdfec0663979b3321a8227b" exitCode=0 Nov 04 01:34:49.429970 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.427508 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2" event={"ID":"4630234e-0999-43f6-9221-007ed3f80001","Type":"ContainerDied","Data":"7f9ad845d074fe5ec22555a7939d0416b5b317108fdfec0663979b3321a8227b"} Nov 04 01:34:49.429970 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.429483 7059 generic.go:356] "Generic (PLEG): container finished" podID="7e4a1625-eab6-4e98-b8e8-0ccc12edc542" containerID="f2681d5814ddc28822bd4a6ca61d61351bcc6b7f07b3632583ad4fbc9de41525" exitCode=0 Nov 04 01:34:49.429970 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.429510 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" event={"ID":"7e4a1625-eab6-4e98-b8e8-0ccc12edc542","Type":"ContainerDied","Data":"f2681d5814ddc28822bd4a6ca61d61351bcc6b7f07b3632583ad4fbc9de41525"} Nov 04 01:34:49.702806 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.702779 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2" Nov 04 01:34:49.776970 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.775781 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:34:49.797322 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.797294 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4630234e-0999-43f6-9221-007ed3f80001-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4630234e-0999-43f6-9221-007ed3f80001-openshift-service-ca\") pod \"4630234e-0999-43f6-9221-007ed3f80001\" (UID: \"4630234e-0999-43f6-9221-007ed3f80001\") " Nov 04 01:34:49.797472 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.797400 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4630234e-0999-43f6-9221-007ed3f80001-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4630234e-0999-43f6-9221-007ed3f80001-metrics-certs\") pod \"4630234e-0999-43f6-9221-007ed3f80001\" (UID: \"4630234e-0999-43f6-9221-007ed3f80001\") " Nov 04 01:34:49.797542 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.797474 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4630234e-0999-43f6-9221-007ed3f80001-kube-api-access-975w2\" (UniqueName: \"kubernetes.io/projected/4630234e-0999-43f6-9221-007ed3f80001-kube-api-access-975w2\") pod \"4630234e-0999-43f6-9221-007ed3f80001\" (UID: \"4630234e-0999-43f6-9221-007ed3f80001\") " Nov 04 01:34:49.797629 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.797605 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4630234e-0999-43f6-9221-007ed3f80001-openshift-service-ca" pod "4630234e-0999-43f6-9221-007ed3f80001" (UID: "4630234e-0999-43f6-9221-007ed3f80001"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:49.797684 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.797653 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4630234e-0999-43f6-9221-007ed3f80001-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4630234e-0999-43f6-9221-007ed3f80001-maas-api-tls\") pod \"4630234e-0999-43f6-9221-007ed3f80001\" (UID: \"4630234e-0999-43f6-9221-007ed3f80001\") " Nov 04 01:34:49.798220 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.798185 7059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4630234e-0999-43f6-9221-007ed3f80001-openshift-service-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:49.800963 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.800701 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4630234e-0999-43f6-9221-007ed3f80001-metrics-certs" pod "4630234e-0999-43f6-9221-007ed3f80001" (UID: "4630234e-0999-43f6-9221-007ed3f80001"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:49.800963 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.800802 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4630234e-0999-43f6-9221-007ed3f80001-maas-api-tls" pod "4630234e-0999-43f6-9221-007ed3f80001" (UID: "4630234e-0999-43f6-9221-007ed3f80001"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:49.801809 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.801548 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4630234e-0999-43f6-9221-007ed3f80001-kube-api-access-975w2" pod "4630234e-0999-43f6-9221-007ed3f80001" (UID: "4630234e-0999-43f6-9221-007ed3f80001"). InnerVolumeSpecName "kube-api-access-975w2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:49.899043 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.898976 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istio-data\") pod \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " Nov 04 01:34:49.899165 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.899065 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istio-podinfo\") pod \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " Nov 04 01:34:49.899165 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.899110 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-kube-api-access-cbmr6\" (UniqueName: \"kubernetes.io/projected/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-kube-api-access-cbmr6\") pod \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " Nov 04 01:34:49.899246 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.899194 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istio-envoy\") pod \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " Nov 04 01:34:49.899246 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.899227 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-workload-socket\") pod \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " Nov 04 01:34:49.899328 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.899239 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istio-data" pod "7e4a1625-eab6-4e98-b8e8-0ccc12edc542" (UID: "7e4a1625-eab6-4e98-b8e8-0ccc12edc542"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:49.899328 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.899252 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-credential-socket\") pod \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " Nov 04 01:34:49.899433 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.899330 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istiod-ca-cert\") pod \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " Nov 04 01:34:49.899433 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.899382 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-workload-certs\") pod \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " Nov 04 01:34:49.899535 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.899480 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-credential-socket" pod "7e4a1625-eab6-4e98-b8e8-0ccc12edc542" (UID: "7e4a1625-eab6-4e98-b8e8-0ccc12edc542"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:49.899588 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.899486 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istio-token\" (UniqueName: \"kubernetes.io/projected/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istio-token\") pod \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\" (UID: \"7e4a1625-eab6-4e98-b8e8-0ccc12edc542\") " Nov 04 01:34:49.899640 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.899591 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-workload-socket" pod "7e4a1625-eab6-4e98-b8e8-0ccc12edc542" (UID: "7e4a1625-eab6-4e98-b8e8-0ccc12edc542"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:49.899824 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.899800 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-workload-certs" pod "7e4a1625-eab6-4e98-b8e8-0ccc12edc542" (UID: "7e4a1625-eab6-4e98-b8e8-0ccc12edc542"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:49.900050 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.899988 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istiod-ca-cert" pod "7e4a1625-eab6-4e98-b8e8-0ccc12edc542" (UID: "7e4a1625-eab6-4e98-b8e8-0ccc12edc542"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:49.900366 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.900342 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:49.900445 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.900372 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:49.900445 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.900388 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:49.900445 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.900402 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:49.900445 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.900416 7059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4630234e-0999-43f6-9221-007ed3f80001-metrics-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:49.900445 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.900430 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-975w2\" (UniqueName: \"kubernetes.io/projected/4630234e-0999-43f6-9221-007ed3f80001-kube-api-access-975w2\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:49.900727 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.900446 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:49.900727 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.900459 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4630234e-0999-43f6-9221-007ed3f80001-maas-api-tls\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:49.901371 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.901345 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-kube-api-access-cbmr6" pod "7e4a1625-eab6-4e98-b8e8-0ccc12edc542" (UID: "7e4a1625-eab6-4e98-b8e8-0ccc12edc542"). InnerVolumeSpecName "kube-api-access-cbmr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:49.901549 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.901531 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istio-podinfo" pod "7e4a1625-eab6-4e98-b8e8-0ccc12edc542" (UID: "7e4a1625-eab6-4e98-b8e8-0ccc12edc542"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:34:49.901631 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.901554 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istio-token" pod "7e4a1625-eab6-4e98-b8e8-0ccc12edc542" (UID: "7e4a1625-eab6-4e98-b8e8-0ccc12edc542"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:49.901631 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:49.901625 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istio-envoy" pod "7e4a1625-eab6-4e98-b8e8-0ccc12edc542" (UID: "7e4a1625-eab6-4e98-b8e8-0ccc12edc542"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:50.002037 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:50.002001 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:50.002037 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:50.002037 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbmr6\" (UniqueName: \"kubernetes.io/projected/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-kube-api-access-cbmr6\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:50.002246 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:50.002049 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:50.002246 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:50.002058 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e4a1625-eab6-4e98-b8e8-0ccc12edc542-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:50.439802 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:50.439772 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" Nov 04 01:34:50.440320 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:50.439801 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn" event={"ID":"7e4a1625-eab6-4e98-b8e8-0ccc12edc542","Type":"ContainerDied","Data":"330de1aa646d646cc6173cc6b8915d3a86b7d65fb11a935f65c3a4522c50c758"} Nov 04 01:34:50.440320 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:50.439876 7059 scope.go:122] "RemoveContainer" containerID="f2681d5814ddc28822bd4a6ca61d61351bcc6b7f07b3632583ad4fbc9de41525" Nov 04 01:34:50.441617 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:50.441595 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2" Nov 04 01:34:50.441759 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:50.441593 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2" event={"ID":"4630234e-0999-43f6-9221-007ed3f80001","Type":"ContainerDied","Data":"f3bac79b43cb06a1125e1c7566c0eb3bade68238e77ba73e52c39f3091be1d36"} Nov 04 01:34:50.452574 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:50.452555 7059 scope.go:122] "RemoveContainer" containerID="7f9ad845d074fe5ec22555a7939d0416b5b317108fdfec0663979b3321a8227b" Nov 04 01:34:50.475162 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:50.475138 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn"] Nov 04 01:34:50.483177 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:50.483154 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-mh7xn"] Nov 04 01:34:50.497667 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:50.497645 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2"] Nov 04 01:34:50.503362 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:50.503340 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dfcebf-6f64c8549d-wvzj2"] Nov 04 01:34:51.178561 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:51.178524 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4630234e-0999-43f6-9221-007ed3f80001" path="/var/lib/kubelet/pods/4630234e-0999-43f6-9221-007ed3f80001/volumes" Nov 04 01:34:51.179232 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:51.179213 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e4a1625-eab6-4e98-b8e8-0ccc12edc542" path="/var/lib/kubelet/pods/7e4a1625-eab6-4e98-b8e8-0ccc12edc542/volumes" Nov 04 01:34:55.792949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:55.789442 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m"] Nov 04 01:34:55.792949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:55.789688 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m" podUID="fc89af7d-03b6-458b-9265-fb7d71459d81" containerName="maas-api" containerID="cri-o://6be37a4ae275545b6e3afe12b537bc242fd2f90fbe2926f29f6bb8adb642af44" gracePeriod=30 Nov 04 01:34:56.226226 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:56.140978 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx"] Nov 04 01:34:56.226226 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:56.141257 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" podUID="e2652bfd-0f50-4f9b-9bc6-1cd0c9167512" containerName="istio-proxy" containerID="cri-o://2fd81e6a3acd8d6f20fd64e0e03d303b1e2988465e961b7b44ebcbfc33c8053f" gracePeriod=30 Nov 04 01:34:56.226226 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:56.182078 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb"] Nov 04 01:34:56.226226 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:56.182340 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" podUID="8368cee7-3404-4ae8-b6de-00b3434472d6" containerName="istio-proxy" containerID="cri-o://1b21a3a4658177e65b799ef632194adf50a8c3e0547850b554001f6aeb89fdd6" gracePeriod=30 Nov 04 01:34:56.659004 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:56.649887 7059 generic.go:356] "Generic (PLEG): container finished" podID="fc89af7d-03b6-458b-9265-fb7d71459d81" containerID="6be37a4ae275545b6e3afe12b537bc242fd2f90fbe2926f29f6bb8adb642af44" exitCode=0 Nov 04 01:34:56.675948 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:56.669512 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m" event={"ID":"fc89af7d-03b6-458b-9265-fb7d71459d81","Type":"ContainerDied","Data":"6be37a4ae275545b6e3afe12b537bc242fd2f90fbe2926f29f6bb8adb642af44"} Nov 04 01:34:56.814140 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:56.813144 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m" Nov 04 01:34:56.907960 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:56.907163 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc89af7d-03b6-458b-9265-fb7d71459d81-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc89af7d-03b6-458b-9265-fb7d71459d81-metrics-certs\") pod \"fc89af7d-03b6-458b-9265-fb7d71459d81\" (UID: \"fc89af7d-03b6-458b-9265-fb7d71459d81\") " Nov 04 01:34:56.907960 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:56.907221 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc89af7d-03b6-458b-9265-fb7d71459d81-kube-api-access-zg64g\" (UniqueName: \"kubernetes.io/projected/fc89af7d-03b6-458b-9265-fb7d71459d81-kube-api-access-zg64g\") pod \"fc89af7d-03b6-458b-9265-fb7d71459d81\" (UID: \"fc89af7d-03b6-458b-9265-fb7d71459d81\") " Nov 04 01:34:56.914680 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:56.914602 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc89af7d-03b6-458b-9265-fb7d71459d81-kube-api-access-zg64g" pod "fc89af7d-03b6-458b-9265-fb7d71459d81" (UID: "fc89af7d-03b6-458b-9265-fb7d71459d81"). InnerVolumeSpecName "kube-api-access-zg64g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:56.915212 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:56.915170 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc89af7d-03b6-458b-9265-fb7d71459d81-metrics-certs" pod "fc89af7d-03b6-458b-9265-fb7d71459d81" (UID: "fc89af7d-03b6-458b-9265-fb7d71459d81"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:57.010233 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:57.008888 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc89af7d-03b6-458b-9265-fb7d71459d81-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc89af7d-03b6-458b-9265-fb7d71459d81-maas-api-tls\") pod \"fc89af7d-03b6-458b-9265-fb7d71459d81\" (UID: \"fc89af7d-03b6-458b-9265-fb7d71459d81\") " Nov 04 01:34:57.010233 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:57.008962 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc89af7d-03b6-458b-9265-fb7d71459d81-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc89af7d-03b6-458b-9265-fb7d71459d81-openshift-service-ca\") pod \"fc89af7d-03b6-458b-9265-fb7d71459d81\" (UID: \"fc89af7d-03b6-458b-9265-fb7d71459d81\") " Nov 04 01:34:57.010233 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:57.009318 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc89af7d-03b6-458b-9265-fb7d71459d81-openshift-service-ca" pod "fc89af7d-03b6-458b-9265-fb7d71459d81" (UID: "fc89af7d-03b6-458b-9265-fb7d71459d81"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:57.010233 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:57.009590 7059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc89af7d-03b6-458b-9265-fb7d71459d81-openshift-service-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:57.010233 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:57.009609 7059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc89af7d-03b6-458b-9265-fb7d71459d81-metrics-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:57.010233 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:57.009618 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg64g\" (UniqueName: \"kubernetes.io/projected/fc89af7d-03b6-458b-9265-fb7d71459d81-kube-api-access-zg64g\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:57.011750 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:57.011730 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc89af7d-03b6-458b-9265-fb7d71459d81-maas-api-tls" pod "fc89af7d-03b6-458b-9265-fb7d71459d81" (UID: "fc89af7d-03b6-458b-9265-fb7d71459d81"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:57.110824 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:57.110792 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc89af7d-03b6-458b-9265-fb7d71459d81-maas-api-tls\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:57.695957 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:57.695589 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m" event={"ID":"fc89af7d-03b6-458b-9265-fb7d71459d81","Type":"ContainerDied","Data":"61de49cdfc87724d129d42288bfe73ac4afd7e8958306192b6b6ecad2e31e484"} Nov 04 01:34:57.695957 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:57.695696 7059 scope.go:122] "RemoveContainer" containerID="6be37a4ae275545b6e3afe12b537bc242fd2f90fbe2926f29f6bb8adb642af44" Nov 04 01:34:57.695957 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:57.695887 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m" Nov 04 01:34:57.715968 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:57.715944 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m"] Nov 04 01:34:57.722291 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:57.722270 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m"] Nov 04 01:34:59.199896 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:59.199860 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc89af7d-03b6-458b-9265-fb7d71459d81" path="/var/lib/kubelet/pods/fc89af7d-03b6-458b-9265-fb7d71459d81/volumes" Nov 04 01:34:59.385187 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:59.385137 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ca55c1-66d7c55b44-mm9qn"] Nov 04 01:34:59.421126 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:59.420679 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ca55c1-5bb65b58bc-j9bkv"] Nov 04 01:34:59.670867 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:34:59.670770 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-ef6cc9-665cd98f7c-bk92m" podUID="fc89af7d-03b6-458b-9265-fb7d71459d81" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.109:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:35:01.744623 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:01.744584 7059 generic.go:356] "Generic (PLEG): container finished" podID="8368cee7-3404-4ae8-b6de-00b3434472d6" containerID="1b21a3a4658177e65b799ef632194adf50a8c3e0547850b554001f6aeb89fdd6" exitCode=0 Nov 04 01:35:01.747400 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:01.744704 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" event={"ID":"8368cee7-3404-4ae8-b6de-00b3434472d6","Type":"ContainerDied","Data":"1b21a3a4658177e65b799ef632194adf50a8c3e0547850b554001f6aeb89fdd6"} Nov 04 01:35:01.753747 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:01.753716 7059 generic.go:356] "Generic (PLEG): container finished" podID="e2652bfd-0f50-4f9b-9bc6-1cd0c9167512" containerID="2fd81e6a3acd8d6f20fd64e0e03d303b1e2988465e961b7b44ebcbfc33c8053f" exitCode=0 Nov 04 01:35:01.754099 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:01.754072 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" event={"ID":"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512","Type":"ContainerDied","Data":"2fd81e6a3acd8d6f20fd64e0e03d303b1e2988465e961b7b44ebcbfc33c8053f"} Nov 04 01:35:01.809237 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:01.809200 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:35:01.866160 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:01.866136 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:35:01.904716 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:01.902991 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8368cee7-3404-4ae8-b6de-00b3434472d6-kube-api-access-k2m84\" (UniqueName: \"kubernetes.io/projected/8368cee7-3404-4ae8-b6de-00b3434472d6-kube-api-access-k2m84\") pod \"8368cee7-3404-4ae8-b6de-00b3434472d6\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " Nov 04 01:35:01.904716 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:01.903073 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-credential-socket\") pod \"8368cee7-3404-4ae8-b6de-00b3434472d6\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " Nov 04 01:35:01.904716 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:01.903121 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8368cee7-3404-4ae8-b6de-00b3434472d6-istio-token\" (UniqueName: \"kubernetes.io/projected/8368cee7-3404-4ae8-b6de-00b3434472d6-istio-token\") pod \"8368cee7-3404-4ae8-b6de-00b3434472d6\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " Nov 04 01:35:01.904716 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:01.903148 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-workload-certs\") pod \"8368cee7-3404-4ae8-b6de-00b3434472d6\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " Nov 04 01:35:01.904716 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:01.903190 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-workload-socket\") pod \"8368cee7-3404-4ae8-b6de-00b3434472d6\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " Nov 04 01:35:01.904716 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:01.903245 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8368cee7-3404-4ae8-b6de-00b3434472d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8368cee7-3404-4ae8-b6de-00b3434472d6-istiod-ca-cert\") pod \"8368cee7-3404-4ae8-b6de-00b3434472d6\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " Nov 04 01:35:01.904716 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:01.903322 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-istio-data\") pod \"8368cee7-3404-4ae8-b6de-00b3434472d6\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " Nov 04 01:35:01.904716 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:01.903385 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-istio-envoy\") pod \"8368cee7-3404-4ae8-b6de-00b3434472d6\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " Nov 04 01:35:01.904716 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:01.903469 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8368cee7-3404-4ae8-b6de-00b3434472d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8368cee7-3404-4ae8-b6de-00b3434472d6-istio-podinfo\") pod \"8368cee7-3404-4ae8-b6de-00b3434472d6\" (UID: \"8368cee7-3404-4ae8-b6de-00b3434472d6\") " Nov 04 01:35:01.906291 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:01.906235 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-workload-certs" pod "8368cee7-3404-4ae8-b6de-00b3434472d6" (UID: "8368cee7-3404-4ae8-b6de-00b3434472d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:01.906501 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:01.906476 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-credential-socket" pod "8368cee7-3404-4ae8-b6de-00b3434472d6" (UID: "8368cee7-3404-4ae8-b6de-00b3434472d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:01.906997 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:01.906972 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-workload-socket" pod "8368cee7-3404-4ae8-b6de-00b3434472d6" (UID: "8368cee7-3404-4ae8-b6de-00b3434472d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:01.907192 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:01.907168 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-istio-data" pod "8368cee7-3404-4ae8-b6de-00b3434472d6" (UID: "8368cee7-3404-4ae8-b6de-00b3434472d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:01.907346 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:01.907326 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8368cee7-3404-4ae8-b6de-00b3434472d6-istiod-ca-cert" pod "8368cee7-3404-4ae8-b6de-00b3434472d6" (UID: "8368cee7-3404-4ae8-b6de-00b3434472d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:01.932751 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:01.932664 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8368cee7-3404-4ae8-b6de-00b3434472d6-istio-token" pod "8368cee7-3404-4ae8-b6de-00b3434472d6" (UID: "8368cee7-3404-4ae8-b6de-00b3434472d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:01.933560 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:01.933531 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-istio-envoy" pod "8368cee7-3404-4ae8-b6de-00b3434472d6" (UID: "8368cee7-3404-4ae8-b6de-00b3434472d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:01.939722 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:01.939698 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8368cee7-3404-4ae8-b6de-00b3434472d6-istio-podinfo" pod "8368cee7-3404-4ae8-b6de-00b3434472d6" (UID: "8368cee7-3404-4ae8-b6de-00b3434472d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:35:01.944569 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:01.944526 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8368cee7-3404-4ae8-b6de-00b3434472d6-kube-api-access-k2m84" pod "8368cee7-3404-4ae8-b6de-00b3434472d6" (UID: "8368cee7-3404-4ae8-b6de-00b3434472d6"). InnerVolumeSpecName "kube-api-access-k2m84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:02.005126 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.005086 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istiod-ca-cert\") pod \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " Nov 04 01:35:02.005322 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.005189 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-kube-api-access-cj8g4\" (UniqueName: \"kubernetes.io/projected/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-kube-api-access-cj8g4\") pod \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " Nov 04 01:35:02.005322 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.005269 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istio-data\") pod \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " Nov 04 01:35:02.005322 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.005302 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istio-envoy\") pod \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " Nov 04 01:35:02.005503 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.005347 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istio-token\" (UniqueName: \"kubernetes.io/projected/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istio-token\") pod \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " Nov 04 01:35:02.005503 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.005445 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-workload-socket\") pod \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " Nov 04 01:35:02.005609 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.005521 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istiod-ca-cert" pod "e2652bfd-0f50-4f9b-9bc6-1cd0c9167512" (UID: "e2652bfd-0f50-4f9b-9bc6-1cd0c9167512"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:02.005669 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.005620 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istio-data" pod "e2652bfd-0f50-4f9b-9bc6-1cd0c9167512" (UID: "e2652bfd-0f50-4f9b-9bc6-1cd0c9167512"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:02.005669 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.005531 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istio-podinfo\") pod \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " Nov 04 01:35:02.005783 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.005712 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-workload-certs\") pod \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " Nov 04 01:35:02.005846 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.005822 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-workload-socket" pod "e2652bfd-0f50-4f9b-9bc6-1cd0c9167512" (UID: "e2652bfd-0f50-4f9b-9bc6-1cd0c9167512"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:02.005846 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.005825 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-credential-socket\") pod \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\" (UID: \"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512\") " Nov 04 01:35:02.006055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.006029 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-credential-socket" pod "e2652bfd-0f50-4f9b-9bc6-1cd0c9167512" (UID: "e2652bfd-0f50-4f9b-9bc6-1cd0c9167512"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:02.006349 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.006120 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-workload-certs" pod "e2652bfd-0f50-4f9b-9bc6-1cd0c9167512" (UID: "e2652bfd-0f50-4f9b-9bc6-1cd0c9167512"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:02.006723 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.006697 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.006827 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.006727 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.006827 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.006744 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.006827 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.006761 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8368cee7-3404-4ae8-b6de-00b3434472d6-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.006827 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.006776 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.006827 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.006789 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.006827 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.006819 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.007217 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.006835 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2m84\" (UniqueName: \"kubernetes.io/projected/8368cee7-3404-4ae8-b6de-00b3434472d6-kube-api-access-k2m84\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.007217 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.006852 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.007217 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.006866 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8368cee7-3404-4ae8-b6de-00b3434472d6-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.007217 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.006880 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.007217 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.006894 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8368cee7-3404-4ae8-b6de-00b3434472d6-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.007217 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.006909 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.007217 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.006953 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8368cee7-3404-4ae8-b6de-00b3434472d6-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.007820 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.007790 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istio-podinfo" pod "e2652bfd-0f50-4f9b-9bc6-1cd0c9167512" (UID: "e2652bfd-0f50-4f9b-9bc6-1cd0c9167512"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:35:02.008031 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.008013 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-kube-api-access-cj8g4" pod "e2652bfd-0f50-4f9b-9bc6-1cd0c9167512" (UID: "e2652bfd-0f50-4f9b-9bc6-1cd0c9167512"). InnerVolumeSpecName "kube-api-access-cj8g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:02.008207 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.008193 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istio-token" pod "e2652bfd-0f50-4f9b-9bc6-1cd0c9167512" (UID: "e2652bfd-0f50-4f9b-9bc6-1cd0c9167512"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:02.008795 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.008776 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istio-envoy" pod "e2652bfd-0f50-4f9b-9bc6-1cd0c9167512" (UID: "e2652bfd-0f50-4f9b-9bc6-1cd0c9167512"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:02.116079 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.114958 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj8g4\" (UniqueName: \"kubernetes.io/projected/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-kube-api-access-cj8g4\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.116079 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.115001 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.116079 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.115017 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.116079 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.115031 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.775947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.775083 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" Nov 04 01:35:02.781953 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.779449 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb" event={"ID":"8368cee7-3404-4ae8-b6de-00b3434472d6","Type":"ContainerDied","Data":"f5f8c7a464c49d82e10be5ee78e1b78b662068831b06afebf55521babafc7dd4"} Nov 04 01:35:02.781953 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.779524 7059 scope.go:122] "RemoveContainer" containerID="1b21a3a4658177e65b799ef632194adf50a8c3e0547850b554001f6aeb89fdd6" Nov 04 01:35:02.804949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.783125 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" event={"ID":"e2652bfd-0f50-4f9b-9bc6-1cd0c9167512","Type":"ContainerDied","Data":"5216a8bbbc45ea0622396944364dc6789d9c5dc482386050a7608b9235c2f6c2"} Nov 04 01:35:02.804949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.783282 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx" Nov 04 01:35:02.888032 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.887987 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb"] Nov 04 01:35:02.928003 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:02.907989 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-hkrmb"] Nov 04 01:35:03.096052 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:03.096011 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx"] Nov 04 01:35:03.578237 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:03.527555 7059 scope.go:122] "RemoveContainer" containerID="2fd81e6a3acd8d6f20fd64e0e03d303b1e2988465e961b7b44ebcbfc33c8053f" Nov 04 01:35:03.578237 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:03.559603 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8368cee7-3404-4ae8-b6de-00b3434472d6" path="/var/lib/kubelet/pods/8368cee7-3404-4ae8-b6de-00b3434472d6/volumes" Nov 04 01:35:03.578237 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:03.560071 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-vgxfx"] Nov 04 01:35:03.840020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:03.837047 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:35:04.003752 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:04.003307 7059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-7rw6b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:35:04.003752 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:04.003377 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" podUID="994ce0d7-9fe2-494e-a915-94fe06da822e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:35:04.053949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:04.025001 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:35:04.191007 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:04.165016 7059 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:35:04.289949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:04.283344 7059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-ddd7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:35:04.289949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:04.283408 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" podUID="26881d61-360e-43bb-83f8-3122e0026e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:35:04.289949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:04.284995 7059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bprgn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:35:04.289949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:04.285058 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" podUID="5e65c0d9-95d9-43a6-b77f-bd41b3fd6383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:35:04.399951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:04.399538 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-ca55c1-66d7c55b44-mm9qn" podUID="9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5" containerName="payload-processing" containerID="cri-o://1be4734bd41614ec07cb044311ed9b6f947a62b4f4b7d3cb7a161d589d06f316" gracePeriod=25 Nov 04 01:35:04.444001 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:04.432598 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ca55c1-5bb65b58bc-j9bkv" podUID="3744cca1-904b-43ac-870b-c07ced58941c" containerName="payload-pre-processing" containerID="cri-o://015382e8cb8510b13cb6f135ba82a1f7ba8d9db491645e3932f0963c7f8cee7d" gracePeriod=25 Nov 04 01:35:05.203695 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:05.203651 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2652bfd-0f50-4f9b-9bc6-1cd0c9167512" path="/var/lib/kubelet/pods/e2652bfd-0f50-4f9b-9bc6-1cd0c9167512/volumes" Nov 04 01:35:05.393223 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:05.392960 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-ca55c1-66d7c55b44-mm9qn" podUID="9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:35:05.393503 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:05.393444 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ca55c1-5bb65b58bc-j9bkv" podUID="3744cca1-904b-43ac-870b-c07ced58941c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:35:06.154488 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:06.154098 7059 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:35:06.154488 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:06.154169 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" podUID="d91e372a-c89d-41f6-a700-8c41a419a293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:35:06.493337 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:35:06.493299 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:35:07.114421 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:07.114387 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v"] Nov 04 01:35:07.114640 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:07.114601 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v" podUID="f303474f-5ca6-484f-92de-b8080ba7bbc5" containerName="maas-api" containerID="cri-o://ee87dbe2789799d1dee26588d2d6eb2c898e5f537a2f0cca8031a2a895907870" gracePeriod=30 Nov 04 01:35:07.420949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:07.408844 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v" Nov 04 01:35:07.543949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:07.539185 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f303474f-5ca6-484f-92de-b8080ba7bbc5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f303474f-5ca6-484f-92de-b8080ba7bbc5-maas-api-tls\") pod \"f303474f-5ca6-484f-92de-b8080ba7bbc5\" (UID: \"f303474f-5ca6-484f-92de-b8080ba7bbc5\") " Nov 04 01:35:07.543949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:07.539361 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f303474f-5ca6-484f-92de-b8080ba7bbc5-kube-api-access-mfn52\" (UniqueName: \"kubernetes.io/projected/f303474f-5ca6-484f-92de-b8080ba7bbc5-kube-api-access-mfn52\") pod \"f303474f-5ca6-484f-92de-b8080ba7bbc5\" (UID: \"f303474f-5ca6-484f-92de-b8080ba7bbc5\") " Nov 04 01:35:07.543949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:07.539427 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f303474f-5ca6-484f-92de-b8080ba7bbc5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f303474f-5ca6-484f-92de-b8080ba7bbc5-metrics-certs\") pod \"f303474f-5ca6-484f-92de-b8080ba7bbc5\" (UID: \"f303474f-5ca6-484f-92de-b8080ba7bbc5\") " Nov 04 01:35:07.543949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:07.539521 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f303474f-5ca6-484f-92de-b8080ba7bbc5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f303474f-5ca6-484f-92de-b8080ba7bbc5-openshift-service-ca\") pod \"f303474f-5ca6-484f-92de-b8080ba7bbc5\" (UID: \"f303474f-5ca6-484f-92de-b8080ba7bbc5\") " Nov 04 01:35:07.543949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:07.540337 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f303474f-5ca6-484f-92de-b8080ba7bbc5-openshift-service-ca" pod "f303474f-5ca6-484f-92de-b8080ba7bbc5" (UID: "f303474f-5ca6-484f-92de-b8080ba7bbc5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:07.553986 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:07.548186 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f303474f-5ca6-484f-92de-b8080ba7bbc5-maas-api-tls" pod "f303474f-5ca6-484f-92de-b8080ba7bbc5" (UID: "f303474f-5ca6-484f-92de-b8080ba7bbc5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:07.553986 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:07.548210 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f303474f-5ca6-484f-92de-b8080ba7bbc5-metrics-certs" pod "f303474f-5ca6-484f-92de-b8080ba7bbc5" (UID: "f303474f-5ca6-484f-92de-b8080ba7bbc5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:07.553986 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:07.550167 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f303474f-5ca6-484f-92de-b8080ba7bbc5-kube-api-access-mfn52" pod "f303474f-5ca6-484f-92de-b8080ba7bbc5" (UID: "f303474f-5ca6-484f-92de-b8080ba7bbc5"). InnerVolumeSpecName "kube-api-access-mfn52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:07.660630 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:07.660100 7059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f303474f-5ca6-484f-92de-b8080ba7bbc5-metrics-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:07.660630 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:07.660143 7059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f303474f-5ca6-484f-92de-b8080ba7bbc5-openshift-service-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:07.660630 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:07.660159 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f303474f-5ca6-484f-92de-b8080ba7bbc5-maas-api-tls\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:07.660630 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:07.660171 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfn52\" (UniqueName: \"kubernetes.io/projected/f303474f-5ca6-484f-92de-b8080ba7bbc5-kube-api-access-mfn52\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:07.958829 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:07.958796 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww"] Nov 04 01:35:07.959120 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:07.959096 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" podUID="d91e372a-c89d-41f6-a700-8c41a419a293" containerName="istio-proxy" containerID="cri-o://44c0dabb823f6fd42a90bf0f8aec1370e34f941f0e9d82ba5c5b39f64a493648" gracePeriod=30 Nov 04 01:35:07.986978 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:07.985717 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z"] Nov 04 01:35:07.986978 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:07.986022 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" containerID="cri-o://f5a574783f45086ddf36f57f3fb0e14bffa14f1ea7a350ab6836b1a563fa2adb" gracePeriod=30 Nov 04 01:35:08.038978 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.037671 7059 generic.go:356] "Generic (PLEG): container finished" podID="f303474f-5ca6-484f-92de-b8080ba7bbc5" containerID="ee87dbe2789799d1dee26588d2d6eb2c898e5f537a2f0cca8031a2a895907870" exitCode=0 Nov 04 01:35:08.038978 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.037809 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v" event={"ID":"f303474f-5ca6-484f-92de-b8080ba7bbc5","Type":"ContainerDied","Data":"ee87dbe2789799d1dee26588d2d6eb2c898e5f537a2f0cca8031a2a895907870"} Nov 04 01:35:08.038978 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.037859 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v" event={"ID":"f303474f-5ca6-484f-92de-b8080ba7bbc5","Type":"ContainerDied","Data":"b1fcd66caf2826530be426758a928156e8e472158958c39e861385c80aeb05ca"} Nov 04 01:35:08.038978 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.037885 7059 scope.go:122] "RemoveContainer" containerID="ee87dbe2789799d1dee26588d2d6eb2c898e5f537a2f0cca8031a2a895907870" Nov 04 01:35:08.038978 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.038048 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v" Nov 04 01:35:08.059804 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.059781 7059 scope.go:122] "RemoveContainer" containerID="ee87dbe2789799d1dee26588d2d6eb2c898e5f537a2f0cca8031a2a895907870" Nov 04 01:35:08.060218 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:35:08.060195 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee87dbe2789799d1dee26588d2d6eb2c898e5f537a2f0cca8031a2a895907870\": container with ID starting with ee87dbe2789799d1dee26588d2d6eb2c898e5f537a2f0cca8031a2a895907870 not found: ID does not exist" containerID="ee87dbe2789799d1dee26588d2d6eb2c898e5f537a2f0cca8031a2a895907870" Nov 04 01:35:08.060314 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.060227 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee87dbe2789799d1dee26588d2d6eb2c898e5f537a2f0cca8031a2a895907870"} err="failed to get container status \"ee87dbe2789799d1dee26588d2d6eb2c898e5f537a2f0cca8031a2a895907870\": rpc error: code = NotFound desc = could not find container \"ee87dbe2789799d1dee26588d2d6eb2c898e5f537a2f0cca8031a2a895907870\": container with ID starting with ee87dbe2789799d1dee26588d2d6eb2c898e5f537a2f0cca8031a2a895907870 not found: ID does not exist" Nov 04 01:35:08.071448 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.071420 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v"] Nov 04 01:35:08.079165 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.079137 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-016d3d-7f4bb6887f-hnv7v"] Nov 04 01:35:08.759637 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.759613 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:35:08.774978 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.774948 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d91e372a-c89d-41f6-a700-8c41a419a293-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91e372a-c89d-41f6-a700-8c41a419a293-istio-podinfo\") pod \"d91e372a-c89d-41f6-a700-8c41a419a293\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " Nov 04 01:35:08.775124 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.775019 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d91e372a-c89d-41f6-a700-8c41a419a293-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91e372a-c89d-41f6-a700-8c41a419a293-istiod-ca-cert\") pod \"d91e372a-c89d-41f6-a700-8c41a419a293\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " Nov 04 01:35:08.775124 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.775091 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-credential-socket\") pod \"d91e372a-c89d-41f6-a700-8c41a419a293\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " Nov 04 01:35:08.775202 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.775150 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-istio-data\") pod \"d91e372a-c89d-41f6-a700-8c41a419a293\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " Nov 04 01:35:08.775314 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.775295 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d91e372a-c89d-41f6-a700-8c41a419a293-kube-api-access-ggn7j\" (UniqueName: \"kubernetes.io/projected/d91e372a-c89d-41f6-a700-8c41a419a293-kube-api-access-ggn7j\") pod \"d91e372a-c89d-41f6-a700-8c41a419a293\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " Nov 04 01:35:08.775369 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.775334 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-istio-envoy\") pod \"d91e372a-c89d-41f6-a700-8c41a419a293\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " Nov 04 01:35:08.775428 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.775372 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-workload-socket\") pod \"d91e372a-c89d-41f6-a700-8c41a419a293\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " Nov 04 01:35:08.775428 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.775364 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-credential-socket" pod "d91e372a-c89d-41f6-a700-8c41a419a293" (UID: "d91e372a-c89d-41f6-a700-8c41a419a293"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:08.775428 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.775424 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d91e372a-c89d-41f6-a700-8c41a419a293-istio-token\" (UniqueName: \"kubernetes.io/projected/d91e372a-c89d-41f6-a700-8c41a419a293-istio-token\") pod \"d91e372a-c89d-41f6-a700-8c41a419a293\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " Nov 04 01:35:08.775590 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.775448 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d91e372a-c89d-41f6-a700-8c41a419a293-istiod-ca-cert" pod "d91e372a-c89d-41f6-a700-8c41a419a293" (UID: "d91e372a-c89d-41f6-a700-8c41a419a293"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:08.775590 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.775473 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-workload-certs\") pod \"d91e372a-c89d-41f6-a700-8c41a419a293\" (UID: \"d91e372a-c89d-41f6-a700-8c41a419a293\") " Nov 04 01:35:08.775702 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.775641 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-workload-socket" pod "d91e372a-c89d-41f6-a700-8c41a419a293" (UID: "d91e372a-c89d-41f6-a700-8c41a419a293"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:08.775772 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.775720 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-workload-certs" pod "d91e372a-c89d-41f6-a700-8c41a419a293" (UID: "d91e372a-c89d-41f6-a700-8c41a419a293"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:08.775772 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.775736 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-istio-data" pod "d91e372a-c89d-41f6-a700-8c41a419a293" (UID: "d91e372a-c89d-41f6-a700-8c41a419a293"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:08.776478 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.776451 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.776584 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.776481 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91e372a-c89d-41f6-a700-8c41a419a293-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.776584 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.776498 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.776584 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.776513 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.776584 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.776529 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.777659 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.777632 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d91e372a-c89d-41f6-a700-8c41a419a293-istio-podinfo" pod "d91e372a-c89d-41f6-a700-8c41a419a293" (UID: "d91e372a-c89d-41f6-a700-8c41a419a293"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:35:08.777815 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.777794 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91e372a-c89d-41f6-a700-8c41a419a293-kube-api-access-ggn7j" pod "d91e372a-c89d-41f6-a700-8c41a419a293" (UID: "d91e372a-c89d-41f6-a700-8c41a419a293"). InnerVolumeSpecName "kube-api-access-ggn7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:08.778226 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.778202 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-istio-envoy" pod "d91e372a-c89d-41f6-a700-8c41a419a293" (UID: "d91e372a-c89d-41f6-a700-8c41a419a293"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:08.778741 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.778725 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91e372a-c89d-41f6-a700-8c41a419a293-istio-token" pod "d91e372a-c89d-41f6-a700-8c41a419a293" (UID: "d91e372a-c89d-41f6-a700-8c41a419a293"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:08.878384 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.878352 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91e372a-c89d-41f6-a700-8c41a419a293-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.878384 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.878385 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggn7j\" (UniqueName: \"kubernetes.io/projected/d91e372a-c89d-41f6-a700-8c41a419a293-kube-api-access-ggn7j\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.878604 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.878400 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91e372a-c89d-41f6-a700-8c41a419a293-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.878604 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.878412 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91e372a-c89d-41f6-a700-8c41a419a293-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.889303 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.889274 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:35:08.979465 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.979393 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-kube-api-access-ftlzm\" (UniqueName: \"kubernetes.io/projected/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-kube-api-access-ftlzm\") pod \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " Nov 04 01:35:08.979465 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.979456 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-workload-certs\") pod \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " Nov 04 01:35:08.979687 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.979482 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istio-token\" (UniqueName: \"kubernetes.io/projected/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istio-token\") pod \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " Nov 04 01:35:08.979687 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.979507 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istio-data\") pod \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " Nov 04 01:35:08.979687 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.979565 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-credential-socket\") pod \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " Nov 04 01:35:08.979687 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.979601 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istio-podinfo\") pod \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " Nov 04 01:35:08.979687 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.979657 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istio-envoy\") pod \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " Nov 04 01:35:08.979989 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.979803 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-workload-certs" pod "81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" (UID: "81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:08.979989 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.979828 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istio-data" pod "81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" (UID: "81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:08.979989 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.979851 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-workload-socket\") pod \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " Nov 04 01:35:08.979989 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.979861 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-credential-socket" pod "81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" (UID: "81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:08.979989 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.979950 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istiod-ca-cert\") pod \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\" (UID: \"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee\") " Nov 04 01:35:08.980276 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.980150 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-workload-socket" pod "81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" (UID: "81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:08.980570 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.980541 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istiod-ca-cert" pod "81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" (UID: "81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:08.981637 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.981607 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.981780 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.981642 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.981780 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.981657 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.981780 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.981670 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.981780 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.981682 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.982135 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.982115 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istio-token" pod "81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" (UID: "81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:08.982195 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.982113 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istio-podinfo" pod "81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" (UID: "81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:35:08.982195 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.982128 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-kube-api-access-ftlzm" pod "81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" (UID: "81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee"). InnerVolumeSpecName "kube-api-access-ftlzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:08.982373 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:08.982356 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istio-envoy" pod "81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" (UID: "81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:09.048146 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:09.048109 7059 generic.go:356] "Generic (PLEG): container finished" podID="d91e372a-c89d-41f6-a700-8c41a419a293" containerID="44c0dabb823f6fd42a90bf0f8aec1370e34f941f0e9d82ba5c5b39f64a493648" exitCode=0 Nov 04 01:35:09.048299 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:09.048158 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" Nov 04 01:35:09.048299 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:09.048187 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" event={"ID":"d91e372a-c89d-41f6-a700-8c41a419a293","Type":"ContainerDied","Data":"44c0dabb823f6fd42a90bf0f8aec1370e34f941f0e9d82ba5c5b39f64a493648"} Nov 04 01:35:09.048299 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:09.048242 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww" event={"ID":"d91e372a-c89d-41f6-a700-8c41a419a293","Type":"ContainerDied","Data":"cb81ee85762c8878a79d5e19b17586d44f7d56df8261998455e958f0daf6e5ad"} Nov 04 01:35:09.048299 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:09.048273 7059 scope.go:122] "RemoveContainer" containerID="44c0dabb823f6fd42a90bf0f8aec1370e34f941f0e9d82ba5c5b39f64a493648" Nov 04 01:35:09.050556 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:09.050531 7059 generic.go:356] "Generic (PLEG): container finished" podID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerID="f5a574783f45086ddf36f57f3fb0e14bffa14f1ea7a350ab6836b1a563fa2adb" exitCode=0 Nov 04 01:35:09.050678 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:09.050561 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" event={"ID":"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee","Type":"ContainerDied","Data":"f5a574783f45086ddf36f57f3fb0e14bffa14f1ea7a350ab6836b1a563fa2adb"} Nov 04 01:35:09.050678 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:09.050594 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" Nov 04 01:35:09.050678 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:09.050615 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z" event={"ID":"81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee","Type":"ContainerDied","Data":"cbd60d368592a9bfc522def33537708fe55607826a51e0ebdb6ba72ed434d3e6"} Nov 04 01:35:09.062062 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:09.062039 7059 scope.go:122] "RemoveContainer" containerID="44c0dabb823f6fd42a90bf0f8aec1370e34f941f0e9d82ba5c5b39f64a493648" Nov 04 01:35:09.062362 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:35:09.062339 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44c0dabb823f6fd42a90bf0f8aec1370e34f941f0e9d82ba5c5b39f64a493648\": container with ID starting with 44c0dabb823f6fd42a90bf0f8aec1370e34f941f0e9d82ba5c5b39f64a493648 not found: ID does not exist" containerID="44c0dabb823f6fd42a90bf0f8aec1370e34f941f0e9d82ba5c5b39f64a493648" Nov 04 01:35:09.062423 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:09.062378 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c0dabb823f6fd42a90bf0f8aec1370e34f941f0e9d82ba5c5b39f64a493648"} err="failed to get container status \"44c0dabb823f6fd42a90bf0f8aec1370e34f941f0e9d82ba5c5b39f64a493648\": rpc error: code = NotFound desc = could not find container \"44c0dabb823f6fd42a90bf0f8aec1370e34f941f0e9d82ba5c5b39f64a493648\": container with ID starting with 44c0dabb823f6fd42a90bf0f8aec1370e34f941f0e9d82ba5c5b39f64a493648 not found: ID does not exist" Nov 04 01:35:09.062423 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:09.062402 7059 scope.go:122] "RemoveContainer" containerID="f5a574783f45086ddf36f57f3fb0e14bffa14f1ea7a350ab6836b1a563fa2adb" Nov 04 01:35:09.073720 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:09.073695 7059 scope.go:122] "RemoveContainer" containerID="9ca62b5f6004c1fd875d6f86a094467c63563609f5130cad39e3064338b9346f" Nov 04 01:35:09.076203 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:09.076179 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww"] Nov 04 01:35:09.079748 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:09.079723 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-24hww"] Nov 04 01:35:09.083410 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:09.083384 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.083494 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:09.083411 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.083494 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:09.083426 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.083494 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:09.083435 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftlzm\" (UniqueName: \"kubernetes.io/projected/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee-kube-api-access-ftlzm\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:09.085674 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:09.085652 7059 scope.go:122] "RemoveContainer" containerID="f5a574783f45086ddf36f57f3fb0e14bffa14f1ea7a350ab6836b1a563fa2adb" Nov 04 01:35:09.085997 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:35:09.085964 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5a574783f45086ddf36f57f3fb0e14bffa14f1ea7a350ab6836b1a563fa2adb\": container with ID starting with f5a574783f45086ddf36f57f3fb0e14bffa14f1ea7a350ab6836b1a563fa2adb not found: ID does not exist" containerID="f5a574783f45086ddf36f57f3fb0e14bffa14f1ea7a350ab6836b1a563fa2adb" Nov 04 01:35:09.086093 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:09.086006 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5a574783f45086ddf36f57f3fb0e14bffa14f1ea7a350ab6836b1a563fa2adb"} err="failed to get container status \"f5a574783f45086ddf36f57f3fb0e14bffa14f1ea7a350ab6836b1a563fa2adb\": rpc error: code = NotFound desc = could not find container \"f5a574783f45086ddf36f57f3fb0e14bffa14f1ea7a350ab6836b1a563fa2adb\": container with ID starting with f5a574783f45086ddf36f57f3fb0e14bffa14f1ea7a350ab6836b1a563fa2adb not found: ID does not exist" Nov 04 01:35:09.086093 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:09.086029 7059 scope.go:122] "RemoveContainer" containerID="9ca62b5f6004c1fd875d6f86a094467c63563609f5130cad39e3064338b9346f" Nov 04 01:35:09.086353 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:35:09.086332 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ca62b5f6004c1fd875d6f86a094467c63563609f5130cad39e3064338b9346f\": container with ID starting with 9ca62b5f6004c1fd875d6f86a094467c63563609f5130cad39e3064338b9346f not found: ID does not exist" containerID="9ca62b5f6004c1fd875d6f86a094467c63563609f5130cad39e3064338b9346f" Nov 04 01:35:09.086455 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:09.086358 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ca62b5f6004c1fd875d6f86a094467c63563609f5130cad39e3064338b9346f"} err="failed to get container status \"9ca62b5f6004c1fd875d6f86a094467c63563609f5130cad39e3064338b9346f\": rpc error: code = NotFound desc = could not find container \"9ca62b5f6004c1fd875d6f86a094467c63563609f5130cad39e3064338b9346f\": container with ID starting with 9ca62b5f6004c1fd875d6f86a094467c63563609f5130cad39e3064338b9346f not found: ID does not exist" Nov 04 01:35:09.091136 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:09.091111 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z"] Nov 04 01:35:09.097078 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:09.097058 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-vbz4z"] Nov 04 01:35:09.177672 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:09.177645 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" path="/var/lib/kubelet/pods/81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee/volumes" Nov 04 01:35:09.178272 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:09.178243 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d91e372a-c89d-41f6-a700-8c41a419a293" path="/var/lib/kubelet/pods/d91e372a-c89d-41f6-a700-8c41a419a293/volumes" Nov 04 01:35:09.178726 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:09.178709 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f303474f-5ca6-484f-92de-b8080ba7bbc5" path="/var/lib/kubelet/pods/f303474f-5ca6-484f-92de-b8080ba7bbc5/volumes" Nov 04 01:35:10.654830 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:10.654797 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn"] Nov 04 01:35:10.655267 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:10.655009 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn" podUID="facc3749-8125-4daa-956e-e521856cc7da" containerName="maas-api" containerID="cri-o://48022948fea01455582e00181bbb3a8a312d106a85f98321c7e39e4d7a06846d" gracePeriod=30 Nov 04 01:35:10.979946 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:10.979904 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn" Nov 04 01:35:11.018576 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:11.018547 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/facc3749-8125-4daa-956e-e521856cc7da-metrics-certs\" (UniqueName: \"kubernetes.io/secret/facc3749-8125-4daa-956e-e521856cc7da-metrics-certs\") pod \"facc3749-8125-4daa-956e-e521856cc7da\" (UID: \"facc3749-8125-4daa-956e-e521856cc7da\") " Nov 04 01:35:11.018823 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:11.018673 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/facc3749-8125-4daa-956e-e521856cc7da-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/facc3749-8125-4daa-956e-e521856cc7da-openshift-service-ca\") pod \"facc3749-8125-4daa-956e-e521856cc7da\" (UID: \"facc3749-8125-4daa-956e-e521856cc7da\") " Nov 04 01:35:11.018997 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:11.018878 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/facc3749-8125-4daa-956e-e521856cc7da-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/facc3749-8125-4daa-956e-e521856cc7da-maas-api-tls\") pod \"facc3749-8125-4daa-956e-e521856cc7da\" (UID: \"facc3749-8125-4daa-956e-e521856cc7da\") " Nov 04 01:35:11.019069 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:11.019047 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/facc3749-8125-4daa-956e-e521856cc7da-kube-api-access-sqggg\" (UniqueName: \"kubernetes.io/projected/facc3749-8125-4daa-956e-e521856cc7da-kube-api-access-sqggg\") pod \"facc3749-8125-4daa-956e-e521856cc7da\" (UID: \"facc3749-8125-4daa-956e-e521856cc7da\") " Nov 04 01:35:11.019125 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:11.019067 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/facc3749-8125-4daa-956e-e521856cc7da-openshift-service-ca" pod "facc3749-8125-4daa-956e-e521856cc7da" (UID: "facc3749-8125-4daa-956e-e521856cc7da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:11.019601 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:11.019576 7059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/facc3749-8125-4daa-956e-e521856cc7da-openshift-service-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:11.022125 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:11.021969 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/facc3749-8125-4daa-956e-e521856cc7da-kube-api-access-sqggg" pod "facc3749-8125-4daa-956e-e521856cc7da" (UID: "facc3749-8125-4daa-956e-e521856cc7da"). InnerVolumeSpecName "kube-api-access-sqggg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:11.023080 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:11.023048 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/facc3749-8125-4daa-956e-e521856cc7da-maas-api-tls" pod "facc3749-8125-4daa-956e-e521856cc7da" (UID: "facc3749-8125-4daa-956e-e521856cc7da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:11.023990 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:11.023955 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/facc3749-8125-4daa-956e-e521856cc7da-metrics-certs" pod "facc3749-8125-4daa-956e-e521856cc7da" (UID: "facc3749-8125-4daa-956e-e521856cc7da"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:11.068689 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:11.068660 7059 generic.go:356] "Generic (PLEG): container finished" podID="facc3749-8125-4daa-956e-e521856cc7da" containerID="48022948fea01455582e00181bbb3a8a312d106a85f98321c7e39e4d7a06846d" exitCode=0 Nov 04 01:35:11.068802 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:11.068714 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn" Nov 04 01:35:11.068802 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:11.068748 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn" event={"ID":"facc3749-8125-4daa-956e-e521856cc7da","Type":"ContainerDied","Data":"48022948fea01455582e00181bbb3a8a312d106a85f98321c7e39e4d7a06846d"} Nov 04 01:35:11.068896 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:11.068807 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn" event={"ID":"facc3749-8125-4daa-956e-e521856cc7da","Type":"ContainerDied","Data":"37ab04886888dab059a554a1163e581285571810525ebb3b10ac33ae6c016fae"} Nov 04 01:35:11.068896 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:11.068831 7059 scope.go:122] "RemoveContainer" containerID="48022948fea01455582e00181bbb3a8a312d106a85f98321c7e39e4d7a06846d" Nov 04 01:35:11.079985 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:11.079970 7059 scope.go:122] "RemoveContainer" containerID="48022948fea01455582e00181bbb3a8a312d106a85f98321c7e39e4d7a06846d" Nov 04 01:35:11.080234 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:35:11.080210 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48022948fea01455582e00181bbb3a8a312d106a85f98321c7e39e4d7a06846d\": container with ID starting with 48022948fea01455582e00181bbb3a8a312d106a85f98321c7e39e4d7a06846d not found: ID does not exist" containerID="48022948fea01455582e00181bbb3a8a312d106a85f98321c7e39e4d7a06846d" Nov 04 01:35:11.080305 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:11.080248 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48022948fea01455582e00181bbb3a8a312d106a85f98321c7e39e4d7a06846d"} err="failed to get container status \"48022948fea01455582e00181bbb3a8a312d106a85f98321c7e39e4d7a06846d\": rpc error: code = NotFound desc = could not find container \"48022948fea01455582e00181bbb3a8a312d106a85f98321c7e39e4d7a06846d\": container with ID starting with 48022948fea01455582e00181bbb3a8a312d106a85f98321c7e39e4d7a06846d not found: ID does not exist" Nov 04 01:35:11.098124 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:11.098103 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn"] Nov 04 01:35:11.114778 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:11.114754 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ca55c1-6d97b7896b-2gbmn"] Nov 04 01:35:11.120641 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:11.120610 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/facc3749-8125-4daa-956e-e521856cc7da-maas-api-tls\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:11.120738 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:11.120646 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqggg\" (UniqueName: \"kubernetes.io/projected/facc3749-8125-4daa-956e-e521856cc7da-kube-api-access-sqggg\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:11.120738 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:11.120664 7059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/facc3749-8125-4daa-956e-e521856cc7da-metrics-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:11.184913 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:11.184878 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="facc3749-8125-4daa-956e-e521856cc7da" path="/var/lib/kubelet/pods/facc3749-8125-4daa-956e-e521856cc7da/volumes" Nov 04 01:35:13.442606 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:13.442567 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g"] Nov 04 01:35:13.451689 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:13.451660 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx"] Nov 04 01:35:14.228478 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.228437 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx"] Nov 04 01:35:14.229357 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229337 7059 cpu_manager.go:414] "containerMap: removing container" podUID="ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7" containerName="maas-api" Nov 04 01:35:14.229421 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229360 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7" containerName="maas-api" Nov 04 01:35:14.229421 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229378 7059 cpu_manager.go:414] "containerMap: removing container" podUID="facc3749-8125-4daa-956e-e521856cc7da" containerName="maas-api" Nov 04 01:35:14.229421 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229382 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="facc3749-8125-4daa-956e-e521856cc7da" containerName="maas-api" Nov 04 01:35:14.229421 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229388 7059 cpu_manager.go:414] "containerMap: removing container" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" Nov 04 01:35:14.229421 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229393 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" Nov 04 01:35:14.229421 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229402 7059 cpu_manager.go:414] "containerMap: removing container" podUID="f303474f-5ca6-484f-92de-b8080ba7bbc5" containerName="maas-api" Nov 04 01:35:14.229421 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229407 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f303474f-5ca6-484f-92de-b8080ba7bbc5" containerName="maas-api" Nov 04 01:35:14.229421 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229415 7059 cpu_manager.go:414] "containerMap: removing container" podUID="3eb77296-8f8c-4ed8-8d70-f6cbdb09b888" containerName="maas-api" Nov 04 01:35:14.229421 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229419 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eb77296-8f8c-4ed8-8d70-f6cbdb09b888" containerName="maas-api" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229428 7059 cpu_manager.go:414] "containerMap: removing container" podUID="142d5751-9524-4f5c-8072-d05d5ca426ad" containerName="payload-pre-processing" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229433 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="142d5751-9524-4f5c-8072-d05d5ca426ad" containerName="payload-pre-processing" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229443 7059 cpu_manager.go:414] "containerMap: removing container" podUID="e2652bfd-0f50-4f9b-9bc6-1cd0c9167512" containerName="istio-proxy" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229448 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2652bfd-0f50-4f9b-9bc6-1cd0c9167512" containerName="istio-proxy" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229456 7059 cpu_manager.go:414] "containerMap: removing container" podUID="f0f8924f-e452-4b2c-8b41-f970764ff963" containerName="istio-proxy" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229461 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0f8924f-e452-4b2c-8b41-f970764ff963" containerName="istio-proxy" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229468 7059 cpu_manager.go:414] "containerMap: removing container" podUID="d75e9829-735d-4f1c-b79b-55b46d5cfc01" containerName="maas-api" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229473 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d75e9829-735d-4f1c-b79b-55b46d5cfc01" containerName="maas-api" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229479 7059 cpu_manager.go:414] "containerMap: removing container" podUID="158365b9-7071-4a6d-b2a4-f6b56d795ec1" containerName="payload-processing" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229484 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="158365b9-7071-4a6d-b2a4-f6b56d795ec1" containerName="payload-processing" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229494 7059 cpu_manager.go:414] "containerMap: removing container" podUID="d91e372a-c89d-41f6-a700-8c41a419a293" containerName="istio-proxy" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229498 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d91e372a-c89d-41f6-a700-8c41a419a293" containerName="istio-proxy" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229509 7059 cpu_manager.go:414] "containerMap: removing container" podUID="d211e9ea-abd4-41d1-8011-e884ad33695f" containerName="istio-proxy" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229514 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d211e9ea-abd4-41d1-8011-e884ad33695f" containerName="istio-proxy" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229524 7059 cpu_manager.go:414] "containerMap: removing container" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229528 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229536 7059 cpu_manager.go:414] "containerMap: removing container" podUID="f9e034fc-c636-44dd-a58c-b8cf6d3a5734" containerName="istio-proxy" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229540 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9e034fc-c636-44dd-a58c-b8cf6d3a5734" containerName="istio-proxy" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229548 7059 cpu_manager.go:414] "containerMap: removing container" podUID="fc89af7d-03b6-458b-9265-fb7d71459d81" containerName="maas-api" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229552 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc89af7d-03b6-458b-9265-fb7d71459d81" containerName="maas-api" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229559 7059 cpu_manager.go:414] "containerMap: removing container" podUID="4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b" containerName="istio-proxy" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229563 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b" containerName="istio-proxy" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229570 7059 cpu_manager.go:414] "containerMap: removing container" podUID="7e4a1625-eab6-4e98-b8e8-0ccc12edc542" containerName="istio-proxy" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229574 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e4a1625-eab6-4e98-b8e8-0ccc12edc542" containerName="istio-proxy" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229579 7059 cpu_manager.go:414] "containerMap: removing container" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229584 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229593 7059 cpu_manager.go:414] "containerMap: removing container" podUID="8368cee7-3404-4ae8-b6de-00b3434472d6" containerName="istio-proxy" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229596 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8368cee7-3404-4ae8-b6de-00b3434472d6" containerName="istio-proxy" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229607 7059 cpu_manager.go:414] "containerMap: removing container" podUID="4630234e-0999-43f6-9221-007ed3f80001" containerName="maas-api" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229611 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4630234e-0999-43f6-9221-007ed3f80001" containerName="maas-api" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229617 7059 cpu_manager.go:414] "containerMap: removing container" podUID="a606a0bb-d6b9-4e97-863e-82c0666282bd" containerName="maas-api" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229621 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a606a0bb-d6b9-4e97-863e-82c0666282bd" containerName="maas-api" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229628 7059 cpu_manager.go:414] "containerMap: removing container" podUID="2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d" containerName="payload-pre-processing" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229633 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d" containerName="payload-pre-processing" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229639 7059 cpu_manager.go:414] "containerMap: removing container" podUID="b3ea0000-873e-4789-8ad1-d182a29d4489" containerName="payload-pre-processing" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229644 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3ea0000-873e-4789-8ad1-d182a29d4489" containerName="payload-pre-processing" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229651 7059 cpu_manager.go:414] "containerMap: removing container" podUID="0085890d-7ef5-4947-975c-440894141023" containerName="istio-proxy" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229655 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0085890d-7ef5-4947-975c-440894141023" containerName="istio-proxy" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229661 7059 cpu_manager.go:414] "containerMap: removing container" podUID="fdb4a298-c600-4362-97a0-8780f0e48532" containerName="maas-api" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229666 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb4a298-c600-4362-97a0-8780f0e48532" containerName="maas-api" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229675 7059 cpu_manager.go:414] "containerMap: removing container" podUID="53fd8082-374a-44b8-bf5d-25b56fdc7c10" containerName="payload-processing" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229679 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53fd8082-374a-44b8-bf5d-25b56fdc7c10" containerName="payload-processing" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229687 7059 cpu_manager.go:414] "containerMap: removing container" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229692 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229698 7059 cpu_manager.go:414] "containerMap: removing container" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229702 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229709 7059 cpu_manager.go:414] "containerMap: removing container" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229713 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229720 7059 cpu_manager.go:414] "containerMap: removing container" podUID="b59977fd-6792-40e9-9070-1f04beb3f0bf" containerName="istio-proxy" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229723 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b59977fd-6792-40e9-9070-1f04beb3f0bf" containerName="istio-proxy" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229729 7059 cpu_manager.go:414] "containerMap: removing container" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" Nov 04 01:35:14.229757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229733 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" Nov 04 01:35:14.232020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229808 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="53fd8082-374a-44b8-bf5d-25b56fdc7c10" containerName="payload-processing" Nov 04 01:35:14.232020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229816 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad50a174-edcc-4898-b072-90d6c9246859" containerName="main" Nov 04 01:35:14.232020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229821 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c795fc5-4986-4f4d-b6f4-00baa7738286" containerName="main" Nov 04 01:35:14.232020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229829 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b51f24e-65d2-40fd-8b78-70ee1a593d00" containerName="main" Nov 04 01:35:14.232020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229835 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b59977fd-6792-40e9-9070-1f04beb3f0bf" containerName="istio-proxy" Nov 04 01:35:14.232020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229840 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="a747fae3-da69-424e-89c2-28fa527999e8" containerName="main" Nov 04 01:35:14.232020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229847 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba4d5980-b4ce-4e3e-b9f3-65f4c871eca7" containerName="maas-api" Nov 04 01:35:14.232020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229852 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="facc3749-8125-4daa-956e-e521856cc7da" containerName="maas-api" Nov 04 01:35:14.232020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229858 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef88c68d-23be-40b4-87b2-dee154f926aa" containerName="main" Nov 04 01:35:14.232020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229864 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f303474f-5ca6-484f-92de-b8080ba7bbc5" containerName="maas-api" Nov 04 01:35:14.232020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229868 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eb77296-8f8c-4ed8-8d70-f6cbdb09b888" containerName="maas-api" Nov 04 01:35:14.232020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229874 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="142d5751-9524-4f5c-8072-d05d5ca426ad" containerName="payload-pre-processing" Nov 04 01:35:14.232020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229879 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2652bfd-0f50-4f9b-9bc6-1cd0c9167512" containerName="istio-proxy" Nov 04 01:35:14.232020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229885 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0f8924f-e452-4b2c-8b41-f970764ff963" containerName="istio-proxy" Nov 04 01:35:14.232020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229890 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d75e9829-735d-4f1c-b79b-55b46d5cfc01" containerName="maas-api" Nov 04 01:35:14.232020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229894 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="158365b9-7071-4a6d-b2a4-f6b56d795ec1" containerName="payload-processing" Nov 04 01:35:14.232020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229900 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d91e372a-c89d-41f6-a700-8c41a419a293" containerName="istio-proxy" Nov 04 01:35:14.232020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229906 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d211e9ea-abd4-41d1-8011-e884ad33695f" containerName="istio-proxy" Nov 04 01:35:14.232020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229911 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" Nov 04 01:35:14.232020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229916 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9e034fc-c636-44dd-a58c-b8cf6d3a5734" containerName="istio-proxy" Nov 04 01:35:14.232020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229948 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc89af7d-03b6-458b-9265-fb7d71459d81" containerName="maas-api" Nov 04 01:35:14.232020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229955 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e69ed7b-5336-45e6-b0ac-ae05e4f1b79b" containerName="istio-proxy" Nov 04 01:35:14.232020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229962 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e4a1625-eab6-4e98-b8e8-0ccc12edc542" containerName="istio-proxy" Nov 04 01:35:14.232020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229967 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="81f30b7f-8888-4ee9-b5a6-e6c49d6ef6ee" containerName="istio-proxy" Nov 04 01:35:14.232020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229974 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8368cee7-3404-4ae8-b6de-00b3434472d6" containerName="istio-proxy" Nov 04 01:35:14.232020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229981 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4630234e-0999-43f6-9221-007ed3f80001" containerName="maas-api" Nov 04 01:35:14.232020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229986 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="a606a0bb-d6b9-4e97-863e-82c0666282bd" containerName="maas-api" Nov 04 01:35:14.232020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229992 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c5b91a9-d24f-4fa6-8df1-cb27aec9d47d" containerName="payload-pre-processing" Nov 04 01:35:14.232020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.229997 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3ea0000-873e-4789-8ad1-d182a29d4489" containerName="payload-pre-processing" Nov 04 01:35:14.232020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.230002 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="0085890d-7ef5-4947-975c-440894141023" containerName="istio-proxy" Nov 04 01:35:14.232020 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.230007 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdb4a298-c600-4362-97a0-8780f0e48532" containerName="maas-api" Nov 04 01:35:14.233261 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.232541 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:14.256840 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.256804 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx"] Nov 04 01:35:14.377353 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.377313 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:14.377552 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.377401 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:14.377552 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.377437 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:14.377552 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.377517 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:14.377664 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.377625 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:14.377718 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.377669 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:14.377776 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.377753 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xstnm\" (UniqueName: \"kubernetes.io/projected/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-kube-api-access-xstnm\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:14.377815 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.377784 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:14.377815 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.377806 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:14.478652 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.478554 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:14.478652 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.478629 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:14.479240 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.478668 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:14.479240 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.478713 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:14.479240 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.478783 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:14.479240 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.478829 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:14.479240 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.478869 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xstnm\" (UniqueName: \"kubernetes.io/projected/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-kube-api-access-xstnm\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:14.479240 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.478895 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:14.479240 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.478944 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:14.479240 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.479209 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:14.479658 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.479329 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:14.479658 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.479477 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:14.479766 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.479661 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:14.480334 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.480309 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:14.481487 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.481462 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:14.481582 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.481560 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:14.487140 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.487117 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xstnm\" (UniqueName: \"kubernetes.io/projected/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-kube-api-access-xstnm\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:14.487457 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.487440 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:14.549152 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.549128 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:14.832365 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.832319 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx"] Nov 04 01:35:14.833965 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:35:14.833902 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7038b92_d7b9_4dda_ace9_e8b92ae50eb0.slice/crio-5e8d53d59ba2210b2920bf30ae015ef85ffd73c928b7f232d6b9f92a630e616e WatchSource:0}: Error finding container 5e8d53d59ba2210b2920bf30ae015ef85ffd73c928b7f232d6b9f92a630e616e: Status 404 returned error can't find the container with id 5e8d53d59ba2210b2920bf30ae015ef85ffd73c928b7f232d6b9f92a630e616e Nov 04 01:35:14.836801 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.836767 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:35:14.836891 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.836847 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:35:14.837029 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:14.836893 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:35:15.103482 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:15.103387 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" event={"ID":"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0","Type":"ContainerStarted","Data":"685ea10e206c084d189dac850f90a10ffd8e543fcd1b2fae0eed6c9b65126b31"} Nov 04 01:35:15.103482 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:15.103456 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" event={"ID":"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0","Type":"ContainerStarted","Data":"5e8d53d59ba2210b2920bf30ae015ef85ffd73c928b7f232d6b9f92a630e616e"} Nov 04 01:35:15.388303 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:15.388182 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ca55c1-5bb65b58bc-j9bkv" podUID="3744cca1-904b-43ac-870b-c07ced58941c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:35:15.388303 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:15.388206 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-ca55c1-66d7c55b44-mm9qn" podUID="9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:35:15.549875 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:15.549845 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:15.551444 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:15.551414 7059 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 04 01:35:15.551584 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:15.551492 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" podUID="d7038b92-d7b9-4dda-ace9-e8b92ae50eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 04 01:35:16.554414 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:16.554368 7059 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 04 01:35:16.555038 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:16.555003 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" podUID="d7038b92-d7b9-4dda-ace9-e8b92ae50eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 04 01:35:17.564970 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:17.556594 7059 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 04 01:35:17.564970 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:17.556650 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" podUID="d7038b92-d7b9-4dda-ace9-e8b92ae50eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 04 01:35:18.562952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:18.558304 7059 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 04 01:35:18.562952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:18.558366 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" podUID="d7038b92-d7b9-4dda-ace9-e8b92ae50eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 04 01:35:19.561847 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:19.561375 7059 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 04 01:35:19.561847 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:19.561442 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" podUID="d7038b92-d7b9-4dda-ace9-e8b92ae50eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 04 01:35:20.550307 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:20.550270 7059 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 04 01:35:20.550597 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:20.550564 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" podUID="d7038b92-d7b9-4dda-ace9-e8b92ae50eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 04 01:35:21.549895 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:21.549847 7059 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 04 01:35:21.550424 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:21.549903 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" podUID="d7038b92-d7b9-4dda-ace9-e8b92ae50eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 04 01:35:22.553542 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:22.553513 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:22.554071 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:22.554046 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:22.554739 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:22.554719 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:22.557491 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:22.557444 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" podStartSLOduration=8.557428966 podStartE2EDuration="8.557428966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:35:15.129161178 +0000 UTC m=+2200.071010094" watchObservedRunningTime="2026-11-04 01:35:22.557428966 +0000 UTC m=+2207.499277869" Nov 04 01:35:22.560868 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:22.560847 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-499ddd-677fd64986-9c76n"] Nov 04 01:35:22.565857 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:22.565832 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-499ddd-68c97cbdc5-57gdf"] Nov 04 01:35:24.678289 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:24.677903 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ca55c1-5bb65b58bc-j9bkv" Nov 04 01:35:24.761108 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:24.761084 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-ca55c1-66d7c55b44-mm9qn" Nov 04 01:35:24.846732 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:24.846702 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5-kube-api-access-7f2ff\" (UniqueName: \"kubernetes.io/projected/9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5-kube-api-access-7f2ff\") pod \"9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5\" (UID: \"9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5\") " Nov 04 01:35:24.846910 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:24.846774 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3744cca1-904b-43ac-870b-c07ced58941c-kube-api-access-h78pt\" (UniqueName: \"kubernetes.io/projected/3744cca1-904b-43ac-870b-c07ced58941c-kube-api-access-h78pt\") pod \"3744cca1-904b-43ac-870b-c07ced58941c\" (UID: \"3744cca1-904b-43ac-870b-c07ced58941c\") " Nov 04 01:35:24.847005 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:24.846944 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3744cca1-904b-43ac-870b-c07ced58941c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3744cca1-904b-43ac-870b-c07ced58941c-plugins-config-volume\") pod \"3744cca1-904b-43ac-870b-c07ced58941c\" (UID: \"3744cca1-904b-43ac-870b-c07ced58941c\") " Nov 04 01:35:24.847105 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:24.847082 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5-plugins-config-volume\") pod \"9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5\" (UID: \"9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5\") " Nov 04 01:35:24.847426 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:24.847400 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3744cca1-904b-43ac-870b-c07ced58941c-plugins-config-volume" pod "3744cca1-904b-43ac-870b-c07ced58941c" (UID: "3744cca1-904b-43ac-870b-c07ced58941c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:24.847714 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:24.847450 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5-plugins-config-volume" pod "9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5" (UID: "9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:24.847829 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:24.847784 7059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5-plugins-config-volume\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:24.847829 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:24.847815 7059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3744cca1-904b-43ac-870b-c07ced58941c-plugins-config-volume\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:24.849283 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:24.849261 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5-kube-api-access-7f2ff" pod "9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5" (UID: "9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5"). InnerVolumeSpecName "kube-api-access-7f2ff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:24.849391 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:24.849377 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3744cca1-904b-43ac-870b-c07ced58941c-kube-api-access-h78pt" pod "3744cca1-904b-43ac-870b-c07ced58941c" (UID: "3744cca1-904b-43ac-870b-c07ced58941c"). InnerVolumeSpecName "kube-api-access-h78pt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:24.949312 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:24.949284 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f2ff\" (UniqueName: \"kubernetes.io/projected/9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5-kube-api-access-7f2ff\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:24.949312 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:24.949310 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h78pt\" (UniqueName: \"kubernetes.io/projected/3744cca1-904b-43ac-870b-c07ced58941c-kube-api-access-h78pt\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:25.191446 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:25.191412 7059 generic.go:356] "Generic (PLEG): container finished" podID="9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5" containerID="1be4734bd41614ec07cb044311ed9b6f947a62b4f4b7d3cb7a161d589d06f316" exitCode=0 Nov 04 01:35:25.191614 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:25.191459 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-ca55c1-66d7c55b44-mm9qn" Nov 04 01:35:25.191614 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:25.191505 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-ca55c1-66d7c55b44-mm9qn" event={"ID":"9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5","Type":"ContainerDied","Data":"1be4734bd41614ec07cb044311ed9b6f947a62b4f4b7d3cb7a161d589d06f316"} Nov 04 01:35:25.191614 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:25.191553 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-ca55c1-66d7c55b44-mm9qn" event={"ID":"9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5","Type":"ContainerDied","Data":"05c7d80305f02937b39d9e1104d7dcb3c684016e1dc7d91261bb3ed68dfe1fa7"} Nov 04 01:35:25.191614 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:25.191576 7059 scope.go:122] "RemoveContainer" containerID="1be4734bd41614ec07cb044311ed9b6f947a62b4f4b7d3cb7a161d589d06f316" Nov 04 01:35:25.193392 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:25.193363 7059 generic.go:356] "Generic (PLEG): container finished" podID="3744cca1-904b-43ac-870b-c07ced58941c" containerID="015382e8cb8510b13cb6f135ba82a1f7ba8d9db491645e3932f0963c7f8cee7d" exitCode=0 Nov 04 01:35:25.193554 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:25.193421 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ca55c1-5bb65b58bc-j9bkv" Nov 04 01:35:25.193628 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:25.193583 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ca55c1-5bb65b58bc-j9bkv" event={"ID":"3744cca1-904b-43ac-870b-c07ced58941c","Type":"ContainerDied","Data":"015382e8cb8510b13cb6f135ba82a1f7ba8d9db491645e3932f0963c7f8cee7d"} Nov 04 01:35:25.193682 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:25.193625 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ca55c1-5bb65b58bc-j9bkv" event={"ID":"3744cca1-904b-43ac-870b-c07ced58941c","Type":"ContainerDied","Data":"67a3136505ed4734cece77181a3e4078eca57c4fc4d5bd248a2bdbd6c800ce4c"} Nov 04 01:35:25.203433 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:25.203411 7059 scope.go:122] "RemoveContainer" containerID="1be4734bd41614ec07cb044311ed9b6f947a62b4f4b7d3cb7a161d589d06f316" Nov 04 01:35:25.203718 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:35:25.203698 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1be4734bd41614ec07cb044311ed9b6f947a62b4f4b7d3cb7a161d589d06f316\": container with ID starting with 1be4734bd41614ec07cb044311ed9b6f947a62b4f4b7d3cb7a161d589d06f316 not found: ID does not exist" containerID="1be4734bd41614ec07cb044311ed9b6f947a62b4f4b7d3cb7a161d589d06f316" Nov 04 01:35:25.203796 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:25.203726 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1be4734bd41614ec07cb044311ed9b6f947a62b4f4b7d3cb7a161d589d06f316"} err="failed to get container status \"1be4734bd41614ec07cb044311ed9b6f947a62b4f4b7d3cb7a161d589d06f316\": rpc error: code = NotFound desc = could not find container \"1be4734bd41614ec07cb044311ed9b6f947a62b4f4b7d3cb7a161d589d06f316\": container with ID starting with 1be4734bd41614ec07cb044311ed9b6f947a62b4f4b7d3cb7a161d589d06f316 not found: ID does not exist" Nov 04 01:35:25.203796 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:25.203747 7059 scope.go:122] "RemoveContainer" containerID="015382e8cb8510b13cb6f135ba82a1f7ba8d9db491645e3932f0963c7f8cee7d" Nov 04 01:35:25.213467 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:25.213438 7059 scope.go:122] "RemoveContainer" containerID="015382e8cb8510b13cb6f135ba82a1f7ba8d9db491645e3932f0963c7f8cee7d" Nov 04 01:35:25.214077 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:35:25.214041 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"015382e8cb8510b13cb6f135ba82a1f7ba8d9db491645e3932f0963c7f8cee7d\": container with ID starting with 015382e8cb8510b13cb6f135ba82a1f7ba8d9db491645e3932f0963c7f8cee7d not found: ID does not exist" containerID="015382e8cb8510b13cb6f135ba82a1f7ba8d9db491645e3932f0963c7f8cee7d" Nov 04 01:35:25.214169 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:25.214078 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"015382e8cb8510b13cb6f135ba82a1f7ba8d9db491645e3932f0963c7f8cee7d"} err="failed to get container status \"015382e8cb8510b13cb6f135ba82a1f7ba8d9db491645e3932f0963c7f8cee7d\": rpc error: code = NotFound desc = could not find container \"015382e8cb8510b13cb6f135ba82a1f7ba8d9db491645e3932f0963c7f8cee7d\": container with ID starting with 015382e8cb8510b13cb6f135ba82a1f7ba8d9db491645e3932f0963c7f8cee7d not found: ID does not exist" Nov 04 01:35:25.216460 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:25.216440 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ca55c1-5bb65b58bc-j9bkv"] Nov 04 01:35:25.224592 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:25.224569 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ca55c1-5bb65b58bc-j9bkv"] Nov 04 01:35:25.234841 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:25.234819 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ca55c1-66d7c55b44-mm9qn"] Nov 04 01:35:25.237305 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:25.237288 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ca55c1-66d7c55b44-mm9qn"] Nov 04 01:35:27.178547 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:27.178509 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3744cca1-904b-43ac-870b-c07ced58941c" path="/var/lib/kubelet/pods/3744cca1-904b-43ac-870b-c07ced58941c/volumes" Nov 04 01:35:27.179145 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:27.179126 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5" path="/var/lib/kubelet/pods/9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5/volumes" Nov 04 01:35:27.561999 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:27.561079 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-499ddd-677fd64986-9c76n" podUID="22a00871-73e9-4c9a-aaa6-9046b1004b9e" containerName="payload-processing" containerID="cri-o://d95133a3271c83c30f652fddc6e5f2669944483122712d26d91d65a6e74d87bd" gracePeriod=25 Nov 04 01:35:27.566554 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:27.566500 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-499ddd-68c97cbdc5-57gdf" podUID="1510cc32-e22b-46f0-acfb-3e2994767889" containerName="payload-pre-processing" containerID="cri-o://b9a842d44d3a0eaa9b17b2883ccb01baaf9bd760bb3cf3f40bd2c6823478ec75" gracePeriod=25 Nov 04 01:35:27.649508 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:27.649476 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d7e45d-78c9c88f96-sggpl"] Nov 04 01:35:27.717488 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:27.701976 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d7e45d-7ff9858d99-qv8gj"] Nov 04 01:35:28.483269 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:28.483209 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" containerID="cri-o://973ec1f5711b762177fbe6e7d554dc51906a91e44d4b6f4465dbb38c41a7c5f3" gracePeriod=45 Nov 04 01:35:28.530950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:28.527626 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" containerID="cri-o://dce536f9fb0717644efe768bb8c2e85ef44ba5ecaa3d562d8853587d858b8bf4" gracePeriod=45 Nov 04 01:35:28.789133 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:28.788652 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": dial tcp 10.133.0.85:8000: connect: connection refused" Nov 04 01:35:29.277955 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:29.274337 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": dial tcp 10.133.0.91:8000: connect: connection refused" Nov 04 01:35:29.855996 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:29.849668 7059 generic.go:356] "Generic (PLEG): container finished" podID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerID="973ec1f5711b762177fbe6e7d554dc51906a91e44d4b6f4465dbb38c41a7c5f3" exitCode=0 Nov 04 01:35:29.855996 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:29.849900 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" event={"ID":"614e0cfe-0032-4a0c-b5cf-560fb67723c1","Type":"ContainerDied","Data":"973ec1f5711b762177fbe6e7d554dc51906a91e44d4b6f4465dbb38c41a7c5f3"} Nov 04 01:35:30.103977 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.103917 7059 generic.go:356] "Generic (PLEG): container finished" podID="001c1106-c03c-4986-8082-cd7f161c8817" containerID="dce536f9fb0717644efe768bb8c2e85ef44ba5ecaa3d562d8853587d858b8bf4" exitCode=0 Nov 04 01:35:30.104368 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.104192 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" event={"ID":"001c1106-c03c-4986-8082-cd7f161c8817","Type":"ContainerDied","Data":"dce536f9fb0717644efe768bb8c2e85ef44ba5ecaa3d562d8853587d858b8bf4"} Nov 04 01:35:30.266298 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.266249 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:35:30.288956 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.283663 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/614e0cfe-0032-4a0c-b5cf-560fb67723c1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/614e0cfe-0032-4a0c-b5cf-560fb67723c1-dshm\") pod \"614e0cfe-0032-4a0c-b5cf-560fb67723c1\" (UID: \"614e0cfe-0032-4a0c-b5cf-560fb67723c1\") " Nov 04 01:35:30.288956 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.283748 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/614e0cfe-0032-4a0c-b5cf-560fb67723c1-tls-certs\" (UniqueName: \"kubernetes.io/secret/614e0cfe-0032-4a0c-b5cf-560fb67723c1-tls-certs\") pod \"614e0cfe-0032-4a0c-b5cf-560fb67723c1\" (UID: \"614e0cfe-0032-4a0c-b5cf-560fb67723c1\") " Nov 04 01:35:30.288956 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.287237 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/614e0cfe-0032-4a0c-b5cf-560fb67723c1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/614e0cfe-0032-4a0c-b5cf-560fb67723c1-model-cache\") pod \"614e0cfe-0032-4a0c-b5cf-560fb67723c1\" (UID: \"614e0cfe-0032-4a0c-b5cf-560fb67723c1\") " Nov 04 01:35:30.288956 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.287282 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/614e0cfe-0032-4a0c-b5cf-560fb67723c1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/614e0cfe-0032-4a0c-b5cf-560fb67723c1-tmp-dir\") pod \"614e0cfe-0032-4a0c-b5cf-560fb67723c1\" (UID: \"614e0cfe-0032-4a0c-b5cf-560fb67723c1\") " Nov 04 01:35:30.288956 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.287440 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/614e0cfe-0032-4a0c-b5cf-560fb67723c1-home\" (UniqueName: \"kubernetes.io/empty-dir/614e0cfe-0032-4a0c-b5cf-560fb67723c1-home\") pod \"614e0cfe-0032-4a0c-b5cf-560fb67723c1\" (UID: \"614e0cfe-0032-4a0c-b5cf-560fb67723c1\") " Nov 04 01:35:30.288956 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.287513 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/614e0cfe-0032-4a0c-b5cf-560fb67723c1-kube-api-access-rg8vv\" (UniqueName: \"kubernetes.io/projected/614e0cfe-0032-4a0c-b5cf-560fb67723c1-kube-api-access-rg8vv\") pod \"614e0cfe-0032-4a0c-b5cf-560fb67723c1\" (UID: \"614e0cfe-0032-4a0c-b5cf-560fb67723c1\") " Nov 04 01:35:30.288956 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.288219 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/614e0cfe-0032-4a0c-b5cf-560fb67723c1-home" pod "614e0cfe-0032-4a0c-b5cf-560fb67723c1" (UID: "614e0cfe-0032-4a0c-b5cf-560fb67723c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:30.288956 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.288715 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/614e0cfe-0032-4a0c-b5cf-560fb67723c1-model-cache" pod "614e0cfe-0032-4a0c-b5cf-560fb67723c1" (UID: "614e0cfe-0032-4a0c-b5cf-560fb67723c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:30.288956 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.288775 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/614e0cfe-0032-4a0c-b5cf-560fb67723c1-tmp-dir" pod "614e0cfe-0032-4a0c-b5cf-560fb67723c1" (UID: "614e0cfe-0032-4a0c-b5cf-560fb67723c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:30.308821 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.290329 7059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/614e0cfe-0032-4a0c-b5cf-560fb67723c1-home\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:30.308821 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.290359 7059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/614e0cfe-0032-4a0c-b5cf-560fb67723c1-model-cache\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:30.308821 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.290409 7059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/614e0cfe-0032-4a0c-b5cf-560fb67723c1-tmp-dir\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:30.308821 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.308639 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/614e0cfe-0032-4a0c-b5cf-560fb67723c1-dshm" pod "614e0cfe-0032-4a0c-b5cf-560fb67723c1" (UID: "614e0cfe-0032-4a0c-b5cf-560fb67723c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:30.308821 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.308758 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/614e0cfe-0032-4a0c-b5cf-560fb67723c1-tls-certs" pod "614e0cfe-0032-4a0c-b5cf-560fb67723c1" (UID: "614e0cfe-0032-4a0c-b5cf-560fb67723c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:30.310627 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.310571 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/614e0cfe-0032-4a0c-b5cf-560fb67723c1-kube-api-access-rg8vv" pod "614e0cfe-0032-4a0c-b5cf-560fb67723c1" (UID: "614e0cfe-0032-4a0c-b5cf-560fb67723c1"). InnerVolumeSpecName "kube-api-access-rg8vv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:30.436011 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.399952 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg8vv\" (UniqueName: \"kubernetes.io/projected/614e0cfe-0032-4a0c-b5cf-560fb67723c1-kube-api-access-rg8vv\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:30.436011 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.399989 7059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/614e0cfe-0032-4a0c-b5cf-560fb67723c1-dshm\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:30.436011 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.400005 7059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/614e0cfe-0032-4a0c-b5cf-560fb67723c1-tls-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:30.533730 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.533695 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:35:30.627491 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.627449 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/001c1106-c03c-4986-8082-cd7f161c8817-kube-api-access-z65cs\" (UniqueName: \"kubernetes.io/projected/001c1106-c03c-4986-8082-cd7f161c8817-kube-api-access-z65cs\") pod \"001c1106-c03c-4986-8082-cd7f161c8817\" (UID: \"001c1106-c03c-4986-8082-cd7f161c8817\") " Nov 04 01:35:30.627709 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.627549 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/001c1106-c03c-4986-8082-cd7f161c8817-dshm\" (UniqueName: \"kubernetes.io/empty-dir/001c1106-c03c-4986-8082-cd7f161c8817-dshm\") pod \"001c1106-c03c-4986-8082-cd7f161c8817\" (UID: \"001c1106-c03c-4986-8082-cd7f161c8817\") " Nov 04 01:35:30.627709 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.627641 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/001c1106-c03c-4986-8082-cd7f161c8817-home\" (UniqueName: \"kubernetes.io/empty-dir/001c1106-c03c-4986-8082-cd7f161c8817-home\") pod \"001c1106-c03c-4986-8082-cd7f161c8817\" (UID: \"001c1106-c03c-4986-8082-cd7f161c8817\") " Nov 04 01:35:30.627816 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.627730 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/001c1106-c03c-4986-8082-cd7f161c8817-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/001c1106-c03c-4986-8082-cd7f161c8817-model-cache\") pod \"001c1106-c03c-4986-8082-cd7f161c8817\" (UID: \"001c1106-c03c-4986-8082-cd7f161c8817\") " Nov 04 01:35:30.627816 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.627803 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/001c1106-c03c-4986-8082-cd7f161c8817-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/001c1106-c03c-4986-8082-cd7f161c8817-tmp-dir\") pod \"001c1106-c03c-4986-8082-cd7f161c8817\" (UID: \"001c1106-c03c-4986-8082-cd7f161c8817\") " Nov 04 01:35:30.627979 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.627826 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/001c1106-c03c-4986-8082-cd7f161c8817-tls-certs\" (UniqueName: \"kubernetes.io/secret/001c1106-c03c-4986-8082-cd7f161c8817-tls-certs\") pod \"001c1106-c03c-4986-8082-cd7f161c8817\" (UID: \"001c1106-c03c-4986-8082-cd7f161c8817\") " Nov 04 01:35:30.628768 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.628709 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/001c1106-c03c-4986-8082-cd7f161c8817-home" pod "001c1106-c03c-4986-8082-cd7f161c8817" (UID: "001c1106-c03c-4986-8082-cd7f161c8817"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:30.628870 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.628766 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/001c1106-c03c-4986-8082-cd7f161c8817-model-cache" pod "001c1106-c03c-4986-8082-cd7f161c8817" (UID: "001c1106-c03c-4986-8082-cd7f161c8817"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:30.629144 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.629105 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/001c1106-c03c-4986-8082-cd7f161c8817-tmp-dir" pod "001c1106-c03c-4986-8082-cd7f161c8817" (UID: "001c1106-c03c-4986-8082-cd7f161c8817"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:30.630545 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.630523 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/001c1106-c03c-4986-8082-cd7f161c8817-tls-certs" pod "001c1106-c03c-4986-8082-cd7f161c8817" (UID: "001c1106-c03c-4986-8082-cd7f161c8817"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:30.630761 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.630741 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/001c1106-c03c-4986-8082-cd7f161c8817-dshm" pod "001c1106-c03c-4986-8082-cd7f161c8817" (UID: "001c1106-c03c-4986-8082-cd7f161c8817"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:30.631170 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.631153 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/001c1106-c03c-4986-8082-cd7f161c8817-kube-api-access-z65cs" pod "001c1106-c03c-4986-8082-cd7f161c8817" (UID: "001c1106-c03c-4986-8082-cd7f161c8817"). InnerVolumeSpecName "kube-api-access-z65cs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:30.730019 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.729940 7059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/001c1106-c03c-4986-8082-cd7f161c8817-tmp-dir\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:30.730019 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.729968 7059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/001c1106-c03c-4986-8082-cd7f161c8817-tls-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:30.730019 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.729979 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z65cs\" (UniqueName: \"kubernetes.io/projected/001c1106-c03c-4986-8082-cd7f161c8817-kube-api-access-z65cs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:30.730019 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.729994 7059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/001c1106-c03c-4986-8082-cd7f161c8817-dshm\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:30.730019 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.730003 7059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/001c1106-c03c-4986-8082-cd7f161c8817-home\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:30.730019 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.730010 7059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/001c1106-c03c-4986-8082-cd7f161c8817-model-cache\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:30.787962 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.787892 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:35:30.788142 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.787997 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:35:30.946709 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.946678 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk"] Nov 04 01:35:30.947243 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.946871 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" podUID="f164affe-f5b7-47dd-a165-2fe6f200caca" containerName="istio-proxy" containerID="cri-o://18e55ff02e993f9aa000a331180e12e8faf85ecf5c7a40239bacd4c6e1daebc1" gracePeriod=30 Nov 04 01:35:30.952653 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.952625 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4"] Nov 04 01:35:30.952821 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.952801 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" podUID="cce88e25-4d81-4b48-a229-b72e308c5f18" containerName="istio-proxy" containerID="cri-o://1b14d352d212126128445d15c69a47096833a482797a15d75416be4c251483c3" gracePeriod=30 Nov 04 01:35:30.965734 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.965710 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx"] Nov 04 01:35:30.965906 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:30.965886 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" podUID="d7038b92-d7b9-4dda-ace9-e8b92ae50eb0" containerName="istio-proxy" containerID="cri-o://685ea10e206c084d189dac850f90a10ffd8e543fcd1b2fae0eed6c9b65126b31" gracePeriod=30 Nov 04 01:35:31.075882 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:31.067211 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Nov 04 01:35:31.075882 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:31.067319 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:35:31.112162 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:31.112126 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" Nov 04 01:35:31.112327 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:31.112159 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx" event={"ID":"001c1106-c03c-4986-8082-cd7f161c8817","Type":"ContainerDied","Data":"421019c7148637aaad49e9de94232351d3d5577eed950710628cdddaa7ddde33"} Nov 04 01:35:31.112327 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:31.112215 7059 scope.go:122] "RemoveContainer" containerID="dce536f9fb0717644efe768bb8c2e85ef44ba5ecaa3d562d8853587d858b8bf4" Nov 04 01:35:31.114264 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:31.114223 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" event={"ID":"614e0cfe-0032-4a0c-b5cf-560fb67723c1","Type":"ContainerDied","Data":"04e47bb1f21bae48e97d50e5d9a87fe8dd40592fdc3640cb1251597757870cf3"} Nov 04 01:35:31.114389 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:31.114308 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g" Nov 04 01:35:31.123996 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:31.123588 7059 scope.go:122] "RemoveContainer" containerID="973ec1f5711b762177fbe6e7d554dc51906a91e44d4b6f4465dbb38c41a7c5f3" Nov 04 01:35:31.142001 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:31.141964 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g"] Nov 04 01:35:31.146093 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:31.146068 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-499ddd/distinct-2-w6-499ddd-kserve-b6fdc6bb-f625g"] Nov 04 01:35:31.160185 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:31.160163 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx"] Nov 04 01:35:31.164771 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:31.164748 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-499ddd/distinct-w6-499ddd-kserve-6fd899dfd5-8p5sx"] Nov 04 01:35:31.177902 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:31.177879 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="001c1106-c03c-4986-8082-cd7f161c8817" path="/var/lib/kubelet/pods/001c1106-c03c-4986-8082-cd7f161c8817/volumes" Nov 04 01:35:31.178347 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:31.178332 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" path="/var/lib/kubelet/pods/614e0cfe-0032-4a0c-b5cf-560fb67723c1/volumes" Nov 04 01:35:32.650215 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:32.650151 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-d7e45d-78c9c88f96-sggpl" podUID="d090174e-7ec3-4c2c-a296-f15f9543ea36" containerName="payload-processing" containerID="cri-o://bc649ae8fe26ee698d8c31f856e194fd2da184ae32004b377d05bc5cf06bd468" gracePeriod=25 Nov 04 01:35:32.703338 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:32.703281 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d7e45d-7ff9858d99-qv8gj" podUID="507f1fa6-dc41-4285-9e97-941eb8e9f525" containerName="payload-pre-processing" containerID="cri-o://b53b2c978346dd9a897ea0aa2ac729d2e5a0924eff3c68090e0b573a5cd6e554" gracePeriod=25 Nov 04 01:35:35.796392 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:35.796356 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6"] Nov 04 01:35:35.805072 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:35.796603 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6" podUID="b9c29ddd-82aa-4855-ba74-fa541b0087d8" containerName="maas-api" containerID="cri-o://619ea51e3ee310ce0c365fead6098790c703811ccd4ad03253cde74cda180a5a" gracePeriod=30 Nov 04 01:35:36.152632 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.152604 7059 generic.go:356] "Generic (PLEG): container finished" podID="f164affe-f5b7-47dd-a165-2fe6f200caca" containerID="18e55ff02e993f9aa000a331180e12e8faf85ecf5c7a40239bacd4c6e1daebc1" exitCode=0 Nov 04 01:35:36.152757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.152690 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" event={"ID":"f164affe-f5b7-47dd-a165-2fe6f200caca","Type":"ContainerDied","Data":"18e55ff02e993f9aa000a331180e12e8faf85ecf5c7a40239bacd4c6e1daebc1"} Nov 04 01:35:36.156915 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.156767 7059 generic.go:356] "Generic (PLEG): container finished" podID="d7038b92-d7b9-4dda-ace9-e8b92ae50eb0" containerID="685ea10e206c084d189dac850f90a10ffd8e543fcd1b2fae0eed6c9b65126b31" exitCode=0 Nov 04 01:35:36.156915 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.156818 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" event={"ID":"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0","Type":"ContainerDied","Data":"685ea10e206c084d189dac850f90a10ffd8e543fcd1b2fae0eed6c9b65126b31"} Nov 04 01:35:36.159080 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.158819 7059 generic.go:356] "Generic (PLEG): container finished" podID="cce88e25-4d81-4b48-a229-b72e308c5f18" containerID="1b14d352d212126128445d15c69a47096833a482797a15d75416be4c251483c3" exitCode=0 Nov 04 01:35:36.159080 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.158909 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" event={"ID":"cce88e25-4d81-4b48-a229-b72e308c5f18","Type":"ContainerDied","Data":"1b14d352d212126128445d15c69a47096833a482797a15d75416be4c251483c3"} Nov 04 01:35:36.161064 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.161039 7059 generic.go:356] "Generic (PLEG): container finished" podID="b9c29ddd-82aa-4855-ba74-fa541b0087d8" containerID="619ea51e3ee310ce0c365fead6098790c703811ccd4ad03253cde74cda180a5a" exitCode=0 Nov 04 01:35:36.161273 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.161193 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6" event={"ID":"b9c29ddd-82aa-4855-ba74-fa541b0087d8","Type":"ContainerDied","Data":"619ea51e3ee310ce0c365fead6098790c703811ccd4ad03253cde74cda180a5a"} Nov 04 01:35:36.161273 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.161240 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6" event={"ID":"b9c29ddd-82aa-4855-ba74-fa541b0087d8","Type":"ContainerDied","Data":"b786b0ec118c43f54120284bc10d764ff6603b0d1f391641c030d0f4f22b5148"} Nov 04 01:35:36.161273 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.161255 7059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b786b0ec118c43f54120284bc10d764ff6603b0d1f391641c030d0f4f22b5148" Nov 04 01:35:36.170779 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.170098 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6" Nov 04 01:35:36.280805 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.280779 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:35:36.309185 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.309123 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:35:36.319450 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.319423 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9c29ddd-82aa-4855-ba74-fa541b0087d8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9c29ddd-82aa-4855-ba74-fa541b0087d8-metrics-certs\") pod \"b9c29ddd-82aa-4855-ba74-fa541b0087d8\" (UID: \"b9c29ddd-82aa-4855-ba74-fa541b0087d8\") " Nov 04 01:35:36.319588 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.319519 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9c29ddd-82aa-4855-ba74-fa541b0087d8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9c29ddd-82aa-4855-ba74-fa541b0087d8-maas-api-tls\") pod \"b9c29ddd-82aa-4855-ba74-fa541b0087d8\" (UID: \"b9c29ddd-82aa-4855-ba74-fa541b0087d8\") " Nov 04 01:35:36.319647 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.319631 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9c29ddd-82aa-4855-ba74-fa541b0087d8-kube-api-access-fqq9k\" (UniqueName: \"kubernetes.io/projected/b9c29ddd-82aa-4855-ba74-fa541b0087d8-kube-api-access-fqq9k\") pod \"b9c29ddd-82aa-4855-ba74-fa541b0087d8\" (UID: \"b9c29ddd-82aa-4855-ba74-fa541b0087d8\") " Nov 04 01:35:36.319700 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.319691 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9c29ddd-82aa-4855-ba74-fa541b0087d8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9c29ddd-82aa-4855-ba74-fa541b0087d8-openshift-service-ca\") pod \"b9c29ddd-82aa-4855-ba74-fa541b0087d8\" (UID: \"b9c29ddd-82aa-4855-ba74-fa541b0087d8\") " Nov 04 01:35:36.320092 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.320063 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9c29ddd-82aa-4855-ba74-fa541b0087d8-openshift-service-ca" pod "b9c29ddd-82aa-4855-ba74-fa541b0087d8" (UID: "b9c29ddd-82aa-4855-ba74-fa541b0087d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:36.322144 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.322119 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9c29ddd-82aa-4855-ba74-fa541b0087d8-metrics-certs" pod "b9c29ddd-82aa-4855-ba74-fa541b0087d8" (UID: "b9c29ddd-82aa-4855-ba74-fa541b0087d8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:36.322261 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.322236 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c29ddd-82aa-4855-ba74-fa541b0087d8-kube-api-access-fqq9k" pod "b9c29ddd-82aa-4855-ba74-fa541b0087d8" (UID: "b9c29ddd-82aa-4855-ba74-fa541b0087d8"). InnerVolumeSpecName "kube-api-access-fqq9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:36.322301 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.322258 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9c29ddd-82aa-4855-ba74-fa541b0087d8-maas-api-tls" pod "b9c29ddd-82aa-4855-ba74-fa541b0087d8" (UID: "b9c29ddd-82aa-4855-ba74-fa541b0087d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:36.346176 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.346148 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:36.423055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.420859 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-istio-data\") pod \"cce88e25-4d81-4b48-a229-b72e308c5f18\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " Nov 04 01:35:36.423055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.420955 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-credential-socket\") pod \"cce88e25-4d81-4b48-a229-b72e308c5f18\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " Nov 04 01:35:36.423055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.420992 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istio-envoy\") pod \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " Nov 04 01:35:36.423055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.421072 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f164affe-f5b7-47dd-a165-2fe6f200caca-istio-token\" (UniqueName: \"kubernetes.io/projected/f164affe-f5b7-47dd-a165-2fe6f200caca-istio-token\") pod \"f164affe-f5b7-47dd-a165-2fe6f200caca\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " Nov 04 01:35:36.423055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.421115 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-credential-socket\") pod \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " Nov 04 01:35:36.423055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.421151 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-credential-socket\") pod \"f164affe-f5b7-47dd-a165-2fe6f200caca\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " Nov 04 01:35:36.423055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.421185 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istio-token\" (UniqueName: \"kubernetes.io/projected/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istio-token\") pod \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " Nov 04 01:35:36.423055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.421228 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-workload-certs\") pod \"cce88e25-4d81-4b48-a229-b72e308c5f18\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " Nov 04 01:35:36.423055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.421284 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-kube-api-access-xstnm\" (UniqueName: \"kubernetes.io/projected/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-kube-api-access-xstnm\") pod \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " Nov 04 01:35:36.423055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.421325 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cce88e25-4d81-4b48-a229-b72e308c5f18-istio-token\" (UniqueName: \"kubernetes.io/projected/cce88e25-4d81-4b48-a229-b72e308c5f18-istio-token\") pod \"cce88e25-4d81-4b48-a229-b72e308c5f18\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " Nov 04 01:35:36.423055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.421394 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-workload-socket\") pod \"cce88e25-4d81-4b48-a229-b72e308c5f18\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " Nov 04 01:35:36.423055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.421436 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cce88e25-4d81-4b48-a229-b72e308c5f18-kube-api-access-l7nkg\" (UniqueName: \"kubernetes.io/projected/cce88e25-4d81-4b48-a229-b72e308c5f18-kube-api-access-l7nkg\") pod \"cce88e25-4d81-4b48-a229-b72e308c5f18\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " Nov 04 01:35:36.423055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.421464 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cce88e25-4d81-4b48-a229-b72e308c5f18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce88e25-4d81-4b48-a229-b72e308c5f18-istio-podinfo\") pod \"cce88e25-4d81-4b48-a229-b72e308c5f18\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " Nov 04 01:35:36.423055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.421519 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f164affe-f5b7-47dd-a165-2fe6f200caca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f164affe-f5b7-47dd-a165-2fe6f200caca-istiod-ca-cert\") pod \"f164affe-f5b7-47dd-a165-2fe6f200caca\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " Nov 04 01:35:36.423055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.421549 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-workload-certs\") pod \"f164affe-f5b7-47dd-a165-2fe6f200caca\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " Nov 04 01:35:36.423055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.421578 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-istio-envoy\") pod \"f164affe-f5b7-47dd-a165-2fe6f200caca\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " Nov 04 01:35:36.423055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.421604 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cce88e25-4d81-4b48-a229-b72e308c5f18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce88e25-4d81-4b48-a229-b72e308c5f18-istiod-ca-cert\") pod \"cce88e25-4d81-4b48-a229-b72e308c5f18\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " Nov 04 01:35:36.423055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.421637 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-istio-envoy\") pod \"cce88e25-4d81-4b48-a229-b72e308c5f18\" (UID: \"cce88e25-4d81-4b48-a229-b72e308c5f18\") " Nov 04 01:35:36.423055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.421684 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-istio-data\") pod \"f164affe-f5b7-47dd-a165-2fe6f200caca\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " Nov 04 01:35:36.423055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.421714 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istio-data\") pod \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " Nov 04 01:35:36.423055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.421746 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f164affe-f5b7-47dd-a165-2fe6f200caca-kube-api-access-8wl6b\" (UniqueName: \"kubernetes.io/projected/f164affe-f5b7-47dd-a165-2fe6f200caca-kube-api-access-8wl6b\") pod \"f164affe-f5b7-47dd-a165-2fe6f200caca\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " Nov 04 01:35:36.423055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.421778 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-workload-certs\") pod \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " Nov 04 01:35:36.423055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.421805 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-workload-socket\") pod \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " Nov 04 01:35:36.423055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.421834 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f164affe-f5b7-47dd-a165-2fe6f200caca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f164affe-f5b7-47dd-a165-2fe6f200caca-istio-podinfo\") pod \"f164affe-f5b7-47dd-a165-2fe6f200caca\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " Nov 04 01:35:36.423055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.421875 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-workload-socket\") pod \"f164affe-f5b7-47dd-a165-2fe6f200caca\" (UID: \"f164affe-f5b7-47dd-a165-2fe6f200caca\") " Nov 04 01:35:36.423055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.421909 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istiod-ca-cert\") pod \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " Nov 04 01:35:36.423055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.421976 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istio-podinfo\") pod \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\" (UID: \"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0\") " Nov 04 01:35:36.423055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.422606 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9c29ddd-82aa-4855-ba74-fa541b0087d8-maas-api-tls\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.423055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.422629 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqq9k\" (UniqueName: \"kubernetes.io/projected/b9c29ddd-82aa-4855-ba74-fa541b0087d8-kube-api-access-fqq9k\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.423055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.422655 7059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9c29ddd-82aa-4855-ba74-fa541b0087d8-openshift-service-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.423055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.422670 7059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9c29ddd-82aa-4855-ba74-fa541b0087d8-metrics-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.425006 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.424014 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-499ddd-68c97cbdc5-57gdf" podUID="1510cc32-e22b-46f0-acfb-3e2994767889" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:35:36.425006 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.424129 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-credential-socket" pod "f164affe-f5b7-47dd-a165-2fe6f200caca" (UID: "f164affe-f5b7-47dd-a165-2fe6f200caca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.425006 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.424363 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-istio-data" pod "cce88e25-4d81-4b48-a229-b72e308c5f18" (UID: "cce88e25-4d81-4b48-a229-b72e308c5f18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.425006 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.424568 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-credential-socket" pod "cce88e25-4d81-4b48-a229-b72e308c5f18" (UID: "cce88e25-4d81-4b48-a229-b72e308c5f18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.425232 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.425011 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-workload-socket" pod "d7038b92-d7b9-4dda-ace9-e8b92ae50eb0" (UID: "d7038b92-d7b9-4dda-ace9-e8b92ae50eb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.425297 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.425259 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-credential-socket" pod "d7038b92-d7b9-4dda-ace9-e8b92ae50eb0" (UID: "d7038b92-d7b9-4dda-ace9-e8b92ae50eb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.425743 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.425719 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istio-data" pod "d7038b92-d7b9-4dda-ace9-e8b92ae50eb0" (UID: "d7038b92-d7b9-4dda-ace9-e8b92ae50eb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.426213 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.426190 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce88e25-4d81-4b48-a229-b72e308c5f18-istiod-ca-cert" pod "cce88e25-4d81-4b48-a229-b72e308c5f18" (UID: "cce88e25-4d81-4b48-a229-b72e308c5f18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:36.426443 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.426367 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-workload-certs" pod "cce88e25-4d81-4b48-a229-b72e308c5f18" (UID: "cce88e25-4d81-4b48-a229-b72e308c5f18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.426544 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.426368 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-workload-certs" pod "d7038b92-d7b9-4dda-ace9-e8b92ae50eb0" (UID: "d7038b92-d7b9-4dda-ace9-e8b92ae50eb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.426630 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.426555 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-istio-data" pod "f164affe-f5b7-47dd-a165-2fe6f200caca" (UID: "f164affe-f5b7-47dd-a165-2fe6f200caca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.426720 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.426568 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-workload-socket" pod "f164affe-f5b7-47dd-a165-2fe6f200caca" (UID: "f164affe-f5b7-47dd-a165-2fe6f200caca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.426835 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.426811 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f164affe-f5b7-47dd-a165-2fe6f200caca-istiod-ca-cert" pod "f164affe-f5b7-47dd-a165-2fe6f200caca" (UID: "f164affe-f5b7-47dd-a165-2fe6f200caca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:36.426913 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.426871 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istiod-ca-cert" pod "d7038b92-d7b9-4dda-ace9-e8b92ae50eb0" (UID: "d7038b92-d7b9-4dda-ace9-e8b92ae50eb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:36.427065 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.427042 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-workload-certs" pod "f164affe-f5b7-47dd-a165-2fe6f200caca" (UID: "f164affe-f5b7-47dd-a165-2fe6f200caca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.427710 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.427689 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-workload-socket" pod "cce88e25-4d81-4b48-a229-b72e308c5f18" (UID: "cce88e25-4d81-4b48-a229-b72e308c5f18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.428547 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.428508 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-499ddd-677fd64986-9c76n" podUID="22a00871-73e9-4c9a-aaa6-9046b1004b9e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:35:36.428727 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.428629 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-istio-envoy" pod "f164affe-f5b7-47dd-a165-2fe6f200caca" (UID: "f164affe-f5b7-47dd-a165-2fe6f200caca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.428833 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.428687 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-kube-api-access-xstnm" pod "d7038b92-d7b9-4dda-ace9-e8b92ae50eb0" (UID: "d7038b92-d7b9-4dda-ace9-e8b92ae50eb0"). InnerVolumeSpecName "kube-api-access-xstnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:36.428885 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.428848 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istio-envoy" pod "d7038b92-d7b9-4dda-ace9-e8b92ae50eb0" (UID: "d7038b92-d7b9-4dda-ace9-e8b92ae50eb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.429795 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.429747 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f164affe-f5b7-47dd-a165-2fe6f200caca-kube-api-access-8wl6b" pod "f164affe-f5b7-47dd-a165-2fe6f200caca" (UID: "f164affe-f5b7-47dd-a165-2fe6f200caca"). InnerVolumeSpecName "kube-api-access-8wl6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:36.429795 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.429781 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istio-token" pod "d7038b92-d7b9-4dda-ace9-e8b92ae50eb0" (UID: "d7038b92-d7b9-4dda-ace9-e8b92ae50eb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:36.430001 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.429831 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istio-podinfo" pod "d7038b92-d7b9-4dda-ace9-e8b92ae50eb0" (UID: "d7038b92-d7b9-4dda-ace9-e8b92ae50eb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:35:36.430001 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.429859 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce88e25-4d81-4b48-a229-b72e308c5f18-istio-token" pod "cce88e25-4d81-4b48-a229-b72e308c5f18" (UID: "cce88e25-4d81-4b48-a229-b72e308c5f18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:36.430216 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.429857 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-istio-envoy" pod "cce88e25-4d81-4b48-a229-b72e308c5f18" (UID: "cce88e25-4d81-4b48-a229-b72e308c5f18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.430216 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.429905 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f164affe-f5b7-47dd-a165-2fe6f200caca-istio-token" pod "f164affe-f5b7-47dd-a165-2fe6f200caca" (UID: "f164affe-f5b7-47dd-a165-2fe6f200caca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:36.430216 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.430016 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cce88e25-4d81-4b48-a229-b72e308c5f18-istio-podinfo" pod "cce88e25-4d81-4b48-a229-b72e308c5f18" (UID: "cce88e25-4d81-4b48-a229-b72e308c5f18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:35:36.431283 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.431258 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce88e25-4d81-4b48-a229-b72e308c5f18-kube-api-access-l7nkg" pod "cce88e25-4d81-4b48-a229-b72e308c5f18" (UID: "cce88e25-4d81-4b48-a229-b72e308c5f18"). InnerVolumeSpecName "kube-api-access-l7nkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:36.431465 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.431447 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f164affe-f5b7-47dd-a165-2fe6f200caca-istio-podinfo" pod "f164affe-f5b7-47dd-a165-2fe6f200caca" (UID: "f164affe-f5b7-47dd-a165-2fe6f200caca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:35:36.486539 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:35:36.486495 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:35:36.524407 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.524378 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.524407 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.524407 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.524649 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.524421 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xstnm\" (UniqueName: \"kubernetes.io/projected/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-kube-api-access-xstnm\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.524649 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.524435 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cce88e25-4d81-4b48-a229-b72e308c5f18-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.524649 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.524449 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.524649 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.524461 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7nkg\" (UniqueName: \"kubernetes.io/projected/cce88e25-4d81-4b48-a229-b72e308c5f18-kube-api-access-l7nkg\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.524649 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.524476 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce88e25-4d81-4b48-a229-b72e308c5f18-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.524649 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.524489 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f164affe-f5b7-47dd-a165-2fe6f200caca-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.524649 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.524501 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.524649 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.524515 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.524649 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.524527 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce88e25-4d81-4b48-a229-b72e308c5f18-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.524649 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.524541 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.524649 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.524552 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.524649 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.524564 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.524649 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.524577 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wl6b\" (UniqueName: \"kubernetes.io/projected/f164affe-f5b7-47dd-a165-2fe6f200caca-kube-api-access-8wl6b\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.524649 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.524590 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.524649 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.524604 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.524649 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.524619 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f164affe-f5b7-47dd-a165-2fe6f200caca-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.524649 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.524633 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.524649 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.524646 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.525256 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.524659 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.525256 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.524672 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.525256 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.524686 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce88e25-4d81-4b48-a229-b72e308c5f18-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.525256 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.524700 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.525256 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.524717 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f164affe-f5b7-47dd-a165-2fe6f200caca-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.525256 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.524731 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.525256 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:36.524746 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f164affe-f5b7-47dd-a165-2fe6f200caca-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:37.173997 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:37.169200 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" Nov 04 01:35:37.173997 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:37.169200 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk" event={"ID":"f164affe-f5b7-47dd-a165-2fe6f200caca","Type":"ContainerDied","Data":"bfa55e4772eab9ec18cea0a0d7c2618fa7fad717fc0644a3195b442e27b0684e"} Nov 04 01:35:37.173997 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:37.169349 7059 scope.go:122] "RemoveContainer" containerID="18e55ff02e993f9aa000a331180e12e8faf85ecf5c7a40239bacd4c6e1daebc1" Nov 04 01:35:37.173997 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:37.172129 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" Nov 04 01:35:37.187953 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:37.178181 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6" Nov 04 01:35:37.187953 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:37.178842 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" Nov 04 01:35:37.225204 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:37.225168 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx" event={"ID":"d7038b92-d7b9-4dda-ace9-e8b92ae50eb0","Type":"ContainerDied","Data":"5e8d53d59ba2210b2920bf30ae015ef85ffd73c928b7f232d6b9f92a630e616e"} Nov 04 01:35:37.231420 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:37.230863 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4" event={"ID":"cce88e25-4d81-4b48-a229-b72e308c5f18","Type":"ContainerDied","Data":"e78517b3f60621c83ee99c992d551c04956e574d7eac747847611a6d79cf09c2"} Nov 04 01:35:37.233468 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:37.230912 7059 scope.go:122] "RemoveContainer" containerID="685ea10e206c084d189dac850f90a10ffd8e543fcd1b2fae0eed6c9b65126b31" Nov 04 01:35:37.271082 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:37.271029 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk"] Nov 04 01:35:37.280853 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:37.280623 7059 scope.go:122] "RemoveContainer" containerID="1b14d352d212126128445d15c69a47096833a482797a15d75416be4c251483c3" Nov 04 01:35:37.286970 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:37.286913 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-xl4tk"] Nov 04 01:35:37.308564 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:37.308537 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6"] Nov 04 01:35:37.321167 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:37.321141 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-499ddd-68775ddf5-4fdc6"] Nov 04 01:35:37.331626 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:37.331605 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx"] Nov 04 01:35:37.334789 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:37.334769 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-2qxdx"] Nov 04 01:35:37.350401 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:37.350371 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4"] Nov 04 01:35:37.356373 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:37.356349 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-gd2q4"] Nov 04 01:35:37.416686 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:37.416657 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-d7e45d-78c9c88f96-sggpl" podUID="d090174e-7ec3-4c2c-a296-f15f9543ea36" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:35:37.417385 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:37.417338 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d7e45d-7ff9858d99-qv8gj" podUID="507f1fa6-dc41-4285-9e97-941eb8e9f525" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:35:39.177498 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:39.177462 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9c29ddd-82aa-4855-ba74-fa541b0087d8" path="/var/lib/kubelet/pods/b9c29ddd-82aa-4855-ba74-fa541b0087d8/volumes" Nov 04 01:35:39.177943 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:39.177912 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cce88e25-4d81-4b48-a229-b72e308c5f18" path="/var/lib/kubelet/pods/cce88e25-4d81-4b48-a229-b72e308c5f18/volumes" Nov 04 01:35:39.178334 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:39.178322 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7038b92-d7b9-4dda-ace9-e8b92ae50eb0" path="/var/lib/kubelet/pods/d7038b92-d7b9-4dda-ace9-e8b92ae50eb0/volumes" Nov 04 01:35:39.179055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:39.179040 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f164affe-f5b7-47dd-a165-2fe6f200caca" path="/var/lib/kubelet/pods/f164affe-f5b7-47dd-a165-2fe6f200caca/volumes" Nov 04 01:35:40.920357 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:40.920321 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6"] Nov 04 01:35:40.920890 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:40.920526 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6" podUID="4016d397-9682-4387-b042-68e1ef09f58b" containerName="maas-api" containerID="cri-o://c736b1ddc66f9d44e5a2d5ced7fc3fc748651ab9b43cac148dc8630a2c52e0f8" gracePeriod=30 Nov 04 01:35:41.206482 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:41.206446 7059 generic.go:356] "Generic (PLEG): container finished" podID="4016d397-9682-4387-b042-68e1ef09f58b" containerID="c736b1ddc66f9d44e5a2d5ced7fc3fc748651ab9b43cac148dc8630a2c52e0f8" exitCode=0 Nov 04 01:35:41.206845 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:41.206820 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6" event={"ID":"4016d397-9682-4387-b042-68e1ef09f58b","Type":"ContainerDied","Data":"c736b1ddc66f9d44e5a2d5ced7fc3fc748651ab9b43cac148dc8630a2c52e0f8"} Nov 04 01:35:41.252394 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:41.252365 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6" Nov 04 01:35:41.290246 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:41.290214 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4016d397-9682-4387-b042-68e1ef09f58b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4016d397-9682-4387-b042-68e1ef09f58b-openshift-service-ca\") pod \"4016d397-9682-4387-b042-68e1ef09f58b\" (UID: \"4016d397-9682-4387-b042-68e1ef09f58b\") " Nov 04 01:35:41.290450 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:41.290313 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4016d397-9682-4387-b042-68e1ef09f58b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4016d397-9682-4387-b042-68e1ef09f58b-metrics-certs\") pod \"4016d397-9682-4387-b042-68e1ef09f58b\" (UID: \"4016d397-9682-4387-b042-68e1ef09f58b\") " Nov 04 01:35:41.290450 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:41.290362 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4016d397-9682-4387-b042-68e1ef09f58b-kube-api-access-fgwrd\" (UniqueName: \"kubernetes.io/projected/4016d397-9682-4387-b042-68e1ef09f58b-kube-api-access-fgwrd\") pod \"4016d397-9682-4387-b042-68e1ef09f58b\" (UID: \"4016d397-9682-4387-b042-68e1ef09f58b\") " Nov 04 01:35:41.290576 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:41.290489 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4016d397-9682-4387-b042-68e1ef09f58b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4016d397-9682-4387-b042-68e1ef09f58b-maas-api-tls\") pod \"4016d397-9682-4387-b042-68e1ef09f58b\" (UID: \"4016d397-9682-4387-b042-68e1ef09f58b\") " Nov 04 01:35:41.290655 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:41.290625 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4016d397-9682-4387-b042-68e1ef09f58b-openshift-service-ca" pod "4016d397-9682-4387-b042-68e1ef09f58b" (UID: "4016d397-9682-4387-b042-68e1ef09f58b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:41.291332 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:41.291304 7059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4016d397-9682-4387-b042-68e1ef09f58b-openshift-service-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:41.292835 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:41.292785 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4016d397-9682-4387-b042-68e1ef09f58b-metrics-certs" pod "4016d397-9682-4387-b042-68e1ef09f58b" (UID: "4016d397-9682-4387-b042-68e1ef09f58b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:41.292960 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:41.292851 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4016d397-9682-4387-b042-68e1ef09f58b-maas-api-tls" pod "4016d397-9682-4387-b042-68e1ef09f58b" (UID: "4016d397-9682-4387-b042-68e1ef09f58b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:41.293212 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:41.293197 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4016d397-9682-4387-b042-68e1ef09f58b-kube-api-access-fgwrd" pod "4016d397-9682-4387-b042-68e1ef09f58b" (UID: "4016d397-9682-4387-b042-68e1ef09f58b"). InnerVolumeSpecName "kube-api-access-fgwrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:41.392214 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:41.392180 7059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4016d397-9682-4387-b042-68e1ef09f58b-metrics-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:41.392214 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:41.392212 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgwrd\" (UniqueName: \"kubernetes.io/projected/4016d397-9682-4387-b042-68e1ef09f58b-kube-api-access-fgwrd\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:41.392214 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:41.392222 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4016d397-9682-4387-b042-68e1ef09f58b-maas-api-tls\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:42.213955 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:42.213909 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6" Nov 04 01:35:42.214358 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:42.213952 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6" event={"ID":"4016d397-9682-4387-b042-68e1ef09f58b","Type":"ContainerDied","Data":"8827b2d79245de054f6d356226acb83976f2f8cd4651f1d90ffdb94aac165936"} Nov 04 01:35:42.214358 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:42.214015 7059 scope.go:122] "RemoveContainer" containerID="c736b1ddc66f9d44e5a2d5ced7fc3fc748651ab9b43cac148dc8630a2c52e0f8" Nov 04 01:35:42.236909 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:42.236884 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6"] Nov 04 01:35:42.240864 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:42.240841 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-d7e45d-84666cc69b-jvsg6"] Nov 04 01:35:43.176970 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:43.176907 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4016d397-9682-4387-b042-68e1ef09f58b" path="/var/lib/kubelet/pods/4016d397-9682-4387-b042-68e1ef09f58b/volumes" Nov 04 01:35:44.225690 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.225660 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg"] Nov 04 01:35:44.226276 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.226258 7059 cpu_manager.go:414] "containerMap: removing container" podUID="d7038b92-d7b9-4dda-ace9-e8b92ae50eb0" containerName="istio-proxy" Nov 04 01:35:44.226276 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.226278 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7038b92-d7b9-4dda-ace9-e8b92ae50eb0" containerName="istio-proxy" Nov 04 01:35:44.226369 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.226299 7059 cpu_manager.go:414] "containerMap: removing container" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" Nov 04 01:35:44.226369 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.226307 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" Nov 04 01:35:44.226369 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.226322 7059 cpu_manager.go:414] "containerMap: removing container" podUID="b9c29ddd-82aa-4855-ba74-fa541b0087d8" containerName="maas-api" Nov 04 01:35:44.226369 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.226328 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9c29ddd-82aa-4855-ba74-fa541b0087d8" containerName="maas-api" Nov 04 01:35:44.226369 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.226342 7059 cpu_manager.go:414] "containerMap: removing container" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" Nov 04 01:35:44.226369 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.226346 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" Nov 04 01:35:44.226369 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.226365 7059 cpu_manager.go:414] "containerMap: removing container" podUID="3744cca1-904b-43ac-870b-c07ced58941c" containerName="payload-pre-processing" Nov 04 01:35:44.226369 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.226371 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3744cca1-904b-43ac-870b-c07ced58941c" containerName="payload-pre-processing" Nov 04 01:35:44.226757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.226389 7059 cpu_manager.go:414] "containerMap: removing container" podUID="f164affe-f5b7-47dd-a165-2fe6f200caca" containerName="istio-proxy" Nov 04 01:35:44.226757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.226396 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f164affe-f5b7-47dd-a165-2fe6f200caca" containerName="istio-proxy" Nov 04 01:35:44.226757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.226411 7059 cpu_manager.go:414] "containerMap: removing container" podUID="9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5" containerName="payload-processing" Nov 04 01:35:44.226757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.226416 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5" containerName="payload-processing" Nov 04 01:35:44.226757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.226424 7059 cpu_manager.go:414] "containerMap: removing container" podUID="cce88e25-4d81-4b48-a229-b72e308c5f18" containerName="istio-proxy" Nov 04 01:35:44.226757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.226428 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce88e25-4d81-4b48-a229-b72e308c5f18" containerName="istio-proxy" Nov 04 01:35:44.226757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.226434 7059 cpu_manager.go:414] "containerMap: removing container" podUID="4016d397-9682-4387-b042-68e1ef09f58b" containerName="maas-api" Nov 04 01:35:44.226757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.226438 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4016d397-9682-4387-b042-68e1ef09f58b" containerName="maas-api" Nov 04 01:35:44.226757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.226553 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9c29ddd-82aa-4855-ba74-fa541b0087d8" containerName="maas-api" Nov 04 01:35:44.226757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.226567 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="001c1106-c03c-4986-8082-cd7f161c8817" containerName="main" Nov 04 01:35:44.226757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.226580 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="3744cca1-904b-43ac-870b-c07ced58941c" containerName="payload-pre-processing" Nov 04 01:35:44.226757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.226591 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="f164affe-f5b7-47dd-a165-2fe6f200caca" containerName="istio-proxy" Nov 04 01:35:44.226757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.226599 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="9798d70c-a8f1-4aa5-ba8e-41f902f4f6e5" containerName="payload-processing" Nov 04 01:35:44.226757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.226604 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="cce88e25-4d81-4b48-a229-b72e308c5f18" containerName="istio-proxy" Nov 04 01:35:44.226757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.226610 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4016d397-9682-4387-b042-68e1ef09f58b" containerName="maas-api" Nov 04 01:35:44.226757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.226616 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7038b92-d7b9-4dda-ace9-e8b92ae50eb0" containerName="istio-proxy" Nov 04 01:35:44.226757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.226623 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="614e0cfe-0032-4a0c-b5cf-560fb67723c1" containerName="main" Nov 04 01:35:44.231451 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.231431 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:44.249107 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.248451 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg"] Nov 04 01:35:44.322997 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.322958 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwgr8\" (UniqueName: \"kubernetes.io/projected/44f12c1b-9d91-4575-973e-cd1aadebf87c-kube-api-access-cwgr8\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:44.323173 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.323008 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f12c1b-9d91-4575-973e-cd1aadebf87c-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:44.323173 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.323114 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:44.323173 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.323147 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f12c1b-9d91-4575-973e-cd1aadebf87c-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:44.323317 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.323221 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44f12c1b-9d91-4575-973e-cd1aadebf87c-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:44.323317 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.323284 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:44.323413 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.323338 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:44.323455 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.323423 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:44.323502 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.323470 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:44.424320 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.424290 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwgr8\" (UniqueName: \"kubernetes.io/projected/44f12c1b-9d91-4575-973e-cd1aadebf87c-kube-api-access-cwgr8\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:44.424485 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.424328 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f12c1b-9d91-4575-973e-cd1aadebf87c-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:44.424485 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.424369 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:44.424485 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.424412 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f12c1b-9d91-4575-973e-cd1aadebf87c-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:44.424485 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.424454 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44f12c1b-9d91-4575-973e-cd1aadebf87c-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:44.424709 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.424501 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:44.424709 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.424581 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:44.424709 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.424613 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:44.424709 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.424688 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:44.424947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.424799 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:44.424947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.424905 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:44.425068 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.424969 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:44.425068 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.425035 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:44.425284 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.425260 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f12c1b-9d91-4575-973e-cd1aadebf87c-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:44.427035 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.427012 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:44.427222 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.427204 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f12c1b-9d91-4575-973e-cd1aadebf87c-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:44.435984 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.435964 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwgr8\" (UniqueName: \"kubernetes.io/projected/44f12c1b-9d91-4575-973e-cd1aadebf87c-kube-api-access-cwgr8\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:44.442414 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.442391 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44f12c1b-9d91-4575-973e-cd1aadebf87c-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:44.552301 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.552212 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:44.892533 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.892465 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg"] Nov 04 01:35:44.927759 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.927177 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:35:44.927759 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.927267 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:35:44.927759 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:44.927315 7059 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 04 01:35:45.277100 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:45.277059 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" event={"ID":"44f12c1b-9d91-4575-973e-cd1aadebf87c","Type":"ContainerStarted","Data":"85598e395bc0caae1a06fc5c10b62b5c1160ebc1170fcd738ca8823169634dcc"} Nov 04 01:35:45.278795 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:45.278770 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" event={"ID":"44f12c1b-9d91-4575-973e-cd1aadebf87c","Type":"ContainerStarted","Data":"6af3eef21a13fe188ef5a3d423f366a4969e09ad5fbb6cc7bb8b0c7fe8ba98b4"} Nov 04 01:35:45.553079 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:45.553005 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:46.407667 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:46.407631 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-499ddd-68c97cbdc5-57gdf" podUID="1510cc32-e22b-46f0-acfb-3e2994767889" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:35:46.408173 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:46.407627 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-499ddd-677fd64986-9c76n" podUID="22a00871-73e9-4c9a-aaa6-9046b1004b9e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:35:46.562984 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:46.561068 7059 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 04 01:35:46.562984 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:46.561132 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" podUID="44f12c1b-9d91-4575-973e-cd1aadebf87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 04 01:35:47.415295 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:47.415254 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-d7e45d-78c9c88f96-sggpl" podUID="d090174e-7ec3-4c2c-a296-f15f9543ea36" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:35:47.415782 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:47.415756 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d7e45d-7ff9858d99-qv8gj" podUID="507f1fa6-dc41-4285-9e97-941eb8e9f525" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:35:47.561950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:47.556468 7059 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 04 01:35:47.561950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:47.556529 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" podUID="44f12c1b-9d91-4575-973e-cd1aadebf87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 04 01:35:47.604764 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:35:47.604509 7059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a00871_73e9_4c9a_aaa6_9046b1004b9e.slice/crio-d95133a3271c83c30f652fddc6e5f2669944483122712d26d91d65a6e74d87bd.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1510cc32_e22b_46f0_acfb_3e2994767889.slice/crio-b9a842d44d3a0eaa9b17b2883ccb01baaf9bd760bb3cf3f40bd2c6823478ec75.scope\": RecentStats: unable to find data in memory cache]" Nov 04 01:35:47.604764 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:35:47.604625 7059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1510cc32_e22b_46f0_acfb_3e2994767889.slice/crio-b9a842d44d3a0eaa9b17b2883ccb01baaf9bd760bb3cf3f40bd2c6823478ec75.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1510cc32_e22b_46f0_acfb_3e2994767889.slice/crio-conmon-b9a842d44d3a0eaa9b17b2883ccb01baaf9bd760bb3cf3f40bd2c6823478ec75.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a00871_73e9_4c9a_aaa6_9046b1004b9e.slice/crio-d95133a3271c83c30f652fddc6e5f2669944483122712d26d91d65a6e74d87bd.scope\": RecentStats: unable to find data in memory cache]" Nov 04 01:35:47.920946 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:47.915787 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-499ddd-677fd64986-9c76n" Nov 04 01:35:47.942008 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:47.938702 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" podStartSLOduration=3.938682246 podStartE2EDuration="3.938682246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:35:45.321982292 +0000 UTC m=+2230.263831196" watchObservedRunningTime="2026-11-04 01:35:47.938682246 +0000 UTC m=+2232.880531150" Nov 04 01:35:47.977735 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:47.977708 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-499ddd-68c97cbdc5-57gdf" Nov 04 01:35:47.992806 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:47.992780 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22a00871-73e9-4c9a-aaa6-9046b1004b9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22a00871-73e9-4c9a-aaa6-9046b1004b9e-plugins-config-volume\") pod \"22a00871-73e9-4c9a-aaa6-9046b1004b9e\" (UID: \"22a00871-73e9-4c9a-aaa6-9046b1004b9e\") " Nov 04 01:35:47.992970 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:47.992897 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22a00871-73e9-4c9a-aaa6-9046b1004b9e-kube-api-access-8zjnp\" (UniqueName: \"kubernetes.io/projected/22a00871-73e9-4c9a-aaa6-9046b1004b9e-kube-api-access-8zjnp\") pod \"22a00871-73e9-4c9a-aaa6-9046b1004b9e\" (UID: \"22a00871-73e9-4c9a-aaa6-9046b1004b9e\") " Nov 04 01:35:47.993138 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:47.993117 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22a00871-73e9-4c9a-aaa6-9046b1004b9e-plugins-config-volume" pod "22a00871-73e9-4c9a-aaa6-9046b1004b9e" (UID: "22a00871-73e9-4c9a-aaa6-9046b1004b9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:47.993266 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:47.993250 7059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22a00871-73e9-4c9a-aaa6-9046b1004b9e-plugins-config-volume\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:47.995093 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:47.995069 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22a00871-73e9-4c9a-aaa6-9046b1004b9e-kube-api-access-8zjnp" pod "22a00871-73e9-4c9a-aaa6-9046b1004b9e" (UID: "22a00871-73e9-4c9a-aaa6-9046b1004b9e"). InnerVolumeSpecName "kube-api-access-8zjnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:48.093994 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:48.093963 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1510cc32-e22b-46f0-acfb-3e2994767889-kube-api-access-m5prz\" (UniqueName: \"kubernetes.io/projected/1510cc32-e22b-46f0-acfb-3e2994767889-kube-api-access-m5prz\") pod \"1510cc32-e22b-46f0-acfb-3e2994767889\" (UID: \"1510cc32-e22b-46f0-acfb-3e2994767889\") " Nov 04 01:35:48.094159 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:48.094132 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1510cc32-e22b-46f0-acfb-3e2994767889-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1510cc32-e22b-46f0-acfb-3e2994767889-plugins-config-volume\") pod \"1510cc32-e22b-46f0-acfb-3e2994767889\" (UID: \"1510cc32-e22b-46f0-acfb-3e2994767889\") " Nov 04 01:35:48.094483 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:48.094459 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1510cc32-e22b-46f0-acfb-3e2994767889-plugins-config-volume" pod "1510cc32-e22b-46f0-acfb-3e2994767889" (UID: "1510cc32-e22b-46f0-acfb-3e2994767889"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:48.094570 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:48.094554 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zjnp\" (UniqueName: \"kubernetes.io/projected/22a00871-73e9-4c9a-aaa6-9046b1004b9e-kube-api-access-8zjnp\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:48.094607 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:48.094573 7059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1510cc32-e22b-46f0-acfb-3e2994767889-plugins-config-volume\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:48.096315 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:48.096296 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1510cc32-e22b-46f0-acfb-3e2994767889-kube-api-access-m5prz" pod "1510cc32-e22b-46f0-acfb-3e2994767889" (UID: "1510cc32-e22b-46f0-acfb-3e2994767889"). InnerVolumeSpecName "kube-api-access-m5prz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:48.195574 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:48.195501 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5prz\" (UniqueName: \"kubernetes.io/projected/1510cc32-e22b-46f0-acfb-3e2994767889-kube-api-access-m5prz\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:48.297094 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:48.297056 7059 generic.go:356] "Generic (PLEG): container finished" podID="1510cc32-e22b-46f0-acfb-3e2994767889" containerID="b9a842d44d3a0eaa9b17b2883ccb01baaf9bd760bb3cf3f40bd2c6823478ec75" exitCode=0 Nov 04 01:35:48.297283 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:48.297271 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-499ddd-68c97cbdc5-57gdf" Nov 04 01:35:48.297658 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:48.297633 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-499ddd-68c97cbdc5-57gdf" event={"ID":"1510cc32-e22b-46f0-acfb-3e2994767889","Type":"ContainerDied","Data":"b9a842d44d3a0eaa9b17b2883ccb01baaf9bd760bb3cf3f40bd2c6823478ec75"} Nov 04 01:35:48.297706 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:48.297688 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-499ddd-68c97cbdc5-57gdf" event={"ID":"1510cc32-e22b-46f0-acfb-3e2994767889","Type":"ContainerDied","Data":"5fe928c771a4d888b4a000b342e004349e19cd720f5db91dd8cc9b4ad3b9417e"} Nov 04 01:35:48.297755 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:48.297710 7059 scope.go:122] "RemoveContainer" containerID="b9a842d44d3a0eaa9b17b2883ccb01baaf9bd760bb3cf3f40bd2c6823478ec75" Nov 04 01:35:48.299221 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:48.299200 7059 generic.go:356] "Generic (PLEG): container finished" podID="22a00871-73e9-4c9a-aaa6-9046b1004b9e" containerID="d95133a3271c83c30f652fddc6e5f2669944483122712d26d91d65a6e74d87bd" exitCode=0 Nov 04 01:35:48.299329 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:48.299244 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-499ddd-677fd64986-9c76n" Nov 04 01:35:48.299329 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:48.299267 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-499ddd-677fd64986-9c76n" event={"ID":"22a00871-73e9-4c9a-aaa6-9046b1004b9e","Type":"ContainerDied","Data":"d95133a3271c83c30f652fddc6e5f2669944483122712d26d91d65a6e74d87bd"} Nov 04 01:35:48.299329 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:48.299294 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-499ddd-677fd64986-9c76n" event={"ID":"22a00871-73e9-4c9a-aaa6-9046b1004b9e","Type":"ContainerDied","Data":"77a4bc64fbd8dd01eeff26cbea3a7d518513f9b48baa0eeeb4bc111f082df62e"} Nov 04 01:35:48.310609 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:48.310591 7059 scope.go:122] "RemoveContainer" containerID="b9a842d44d3a0eaa9b17b2883ccb01baaf9bd760bb3cf3f40bd2c6823478ec75" Nov 04 01:35:48.310879 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:35:48.310859 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9a842d44d3a0eaa9b17b2883ccb01baaf9bd760bb3cf3f40bd2c6823478ec75\": container with ID starting with b9a842d44d3a0eaa9b17b2883ccb01baaf9bd760bb3cf3f40bd2c6823478ec75 not found: ID does not exist" containerID="b9a842d44d3a0eaa9b17b2883ccb01baaf9bd760bb3cf3f40bd2c6823478ec75" Nov 04 01:35:48.310961 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:48.310885 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9a842d44d3a0eaa9b17b2883ccb01baaf9bd760bb3cf3f40bd2c6823478ec75"} err="failed to get container status \"b9a842d44d3a0eaa9b17b2883ccb01baaf9bd760bb3cf3f40bd2c6823478ec75\": rpc error: code = NotFound desc = could not find container \"b9a842d44d3a0eaa9b17b2883ccb01baaf9bd760bb3cf3f40bd2c6823478ec75\": container with ID starting with b9a842d44d3a0eaa9b17b2883ccb01baaf9bd760bb3cf3f40bd2c6823478ec75 not found: ID does not exist" Nov 04 01:35:48.310961 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:48.310901 7059 scope.go:122] "RemoveContainer" containerID="d95133a3271c83c30f652fddc6e5f2669944483122712d26d91d65a6e74d87bd" Nov 04 01:35:48.319638 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:48.319546 7059 scope.go:122] "RemoveContainer" containerID="d95133a3271c83c30f652fddc6e5f2669944483122712d26d91d65a6e74d87bd" Nov 04 01:35:48.319823 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:35:48.319804 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d95133a3271c83c30f652fddc6e5f2669944483122712d26d91d65a6e74d87bd\": container with ID starting with d95133a3271c83c30f652fddc6e5f2669944483122712d26d91d65a6e74d87bd not found: ID does not exist" containerID="d95133a3271c83c30f652fddc6e5f2669944483122712d26d91d65a6e74d87bd" Nov 04 01:35:48.319898 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:48.319827 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d95133a3271c83c30f652fddc6e5f2669944483122712d26d91d65a6e74d87bd"} err="failed to get container status \"d95133a3271c83c30f652fddc6e5f2669944483122712d26d91d65a6e74d87bd\": rpc error: code = NotFound desc = could not find container \"d95133a3271c83c30f652fddc6e5f2669944483122712d26d91d65a6e74d87bd\": container with ID starting with d95133a3271c83c30f652fddc6e5f2669944483122712d26d91d65a6e74d87bd not found: ID does not exist" Nov 04 01:35:48.323607 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:48.323584 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-499ddd-677fd64986-9c76n"] Nov 04 01:35:48.327911 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:48.327890 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-499ddd-677fd64986-9c76n"] Nov 04 01:35:48.363276 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:48.363227 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-499ddd-68c97cbdc5-57gdf"] Nov 04 01:35:48.366500 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:48.366478 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-499ddd-68c97cbdc5-57gdf"] Nov 04 01:35:48.553402 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:48.553317 7059 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 04 01:35:48.563065 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:48.563015 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" podUID="44f12c1b-9d91-4575-973e-cd1aadebf87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 04 01:35:49.177724 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:49.177691 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1510cc32-e22b-46f0-acfb-3e2994767889" path="/var/lib/kubelet/pods/1510cc32-e22b-46f0-acfb-3e2994767889/volumes" Nov 04 01:35:49.178264 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:49.178245 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22a00871-73e9-4c9a-aaa6-9046b1004b9e" path="/var/lib/kubelet/pods/22a00871-73e9-4c9a-aaa6-9046b1004b9e/volumes" Nov 04 01:35:49.563011 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:49.560289 7059 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 04 01:35:49.563011 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:49.560367 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" podUID="44f12c1b-9d91-4575-973e-cd1aadebf87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 04 01:35:50.559448 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:50.559409 7059 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 04 01:35:50.559721 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:50.559690 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" podUID="44f12c1b-9d91-4575-973e-cd1aadebf87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 04 01:35:51.552947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:51.552890 7059 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 04 01:35:51.553334 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:51.552965 7059 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" podUID="44f12c1b-9d91-4575-973e-cd1aadebf87c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 04 01:35:52.557108 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:52.557078 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:52.557528 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:52.557367 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:52.558181 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:52.558156 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:52.973799 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:52.973769 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-d7e45d-78c9c88f96-sggpl" Nov 04 01:35:53.046649 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.046607 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d7e45d-7ff9858d99-qv8gj" Nov 04 01:35:53.061600 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.061515 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d090174e-7ec3-4c2c-a296-f15f9543ea36-kube-api-access-9vbj7\" (UniqueName: \"kubernetes.io/projected/d090174e-7ec3-4c2c-a296-f15f9543ea36-kube-api-access-9vbj7\") pod \"d090174e-7ec3-4c2c-a296-f15f9543ea36\" (UID: \"d090174e-7ec3-4c2c-a296-f15f9543ea36\") " Nov 04 01:35:53.061822 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.061796 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d090174e-7ec3-4c2c-a296-f15f9543ea36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d090174e-7ec3-4c2c-a296-f15f9543ea36-plugins-config-volume\") pod \"d090174e-7ec3-4c2c-a296-f15f9543ea36\" (UID: \"d090174e-7ec3-4c2c-a296-f15f9543ea36\") " Nov 04 01:35:53.062712 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.062681 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d090174e-7ec3-4c2c-a296-f15f9543ea36-plugins-config-volume" pod "d090174e-7ec3-4c2c-a296-f15f9543ea36" (UID: "d090174e-7ec3-4c2c-a296-f15f9543ea36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:53.064684 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.064654 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d090174e-7ec3-4c2c-a296-f15f9543ea36-kube-api-access-9vbj7" pod "d090174e-7ec3-4c2c-a296-f15f9543ea36" (UID: "d090174e-7ec3-4c2c-a296-f15f9543ea36"). InnerVolumeSpecName "kube-api-access-9vbj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:53.163450 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.163414 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/507f1fa6-dc41-4285-9e97-941eb8e9f525-kube-api-access-nwnf9\" (UniqueName: \"kubernetes.io/projected/507f1fa6-dc41-4285-9e97-941eb8e9f525-kube-api-access-nwnf9\") pod \"507f1fa6-dc41-4285-9e97-941eb8e9f525\" (UID: \"507f1fa6-dc41-4285-9e97-941eb8e9f525\") " Nov 04 01:35:53.163614 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.163467 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/507f1fa6-dc41-4285-9e97-941eb8e9f525-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/507f1fa6-dc41-4285-9e97-941eb8e9f525-plugins-config-volume\") pod \"507f1fa6-dc41-4285-9e97-941eb8e9f525\" (UID: \"507f1fa6-dc41-4285-9e97-941eb8e9f525\") " Nov 04 01:35:53.163866 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.163846 7059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d090174e-7ec3-4c2c-a296-f15f9543ea36-plugins-config-volume\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:53.163866 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.163865 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vbj7\" (UniqueName: \"kubernetes.io/projected/d090174e-7ec3-4c2c-a296-f15f9543ea36-kube-api-access-9vbj7\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:53.164066 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.163950 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/507f1fa6-dc41-4285-9e97-941eb8e9f525-plugins-config-volume" pod "507f1fa6-dc41-4285-9e97-941eb8e9f525" (UID: "507f1fa6-dc41-4285-9e97-941eb8e9f525"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:53.165625 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.165603 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/507f1fa6-dc41-4285-9e97-941eb8e9f525-kube-api-access-nwnf9" pod "507f1fa6-dc41-4285-9e97-941eb8e9f525" (UID: "507f1fa6-dc41-4285-9e97-941eb8e9f525"). InnerVolumeSpecName "kube-api-access-nwnf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:53.265432 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.265392 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwnf9\" (UniqueName: \"kubernetes.io/projected/507f1fa6-dc41-4285-9e97-941eb8e9f525-kube-api-access-nwnf9\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:53.265432 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.265433 7059 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/507f1fa6-dc41-4285-9e97-941eb8e9f525-plugins-config-volume\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:53.278672 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.278633 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg"] Nov 04 01:35:53.288075 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.288047 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw"] Nov 04 01:35:53.288350 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.288304 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" podUID="5b925870-1345-4a77-8f3b-2175eb9bb93a" containerName="istio-proxy" containerID="cri-o://8da6d1bb6c9da5ab033ee837b326722c8fb4bca4bf7b5c0bf79cc1cce8fa7bda" gracePeriod=30 Nov 04 01:35:53.309182 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.309153 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq"] Nov 04 01:35:53.309402 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.309381 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" podUID="bf384bd4-f417-42a3-b8a2-cddcbddc8d8b" containerName="istio-proxy" containerID="cri-o://489d393bc645ccd2293f47d7ecd79b0f7743b38ae03b8d48cc61b4e08eb924a8" gracePeriod=30 Nov 04 01:35:53.332820 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.332747 7059 generic.go:356] "Generic (PLEG): container finished" podID="d090174e-7ec3-4c2c-a296-f15f9543ea36" containerID="bc649ae8fe26ee698d8c31f856e194fd2da184ae32004b377d05bc5cf06bd468" exitCode=0 Nov 04 01:35:53.332820 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.332801 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-d7e45d-78c9c88f96-sggpl" Nov 04 01:35:53.332990 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.332830 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-d7e45d-78c9c88f96-sggpl" event={"ID":"d090174e-7ec3-4c2c-a296-f15f9543ea36","Type":"ContainerDied","Data":"bc649ae8fe26ee698d8c31f856e194fd2da184ae32004b377d05bc5cf06bd468"} Nov 04 01:35:53.332990 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.332886 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-d7e45d-78c9c88f96-sggpl" event={"ID":"d090174e-7ec3-4c2c-a296-f15f9543ea36","Type":"ContainerDied","Data":"9597bb7f4eafa2b249841aa5f842c4e1f280c2d58f453ed73c10309350d230f9"} Nov 04 01:35:53.332990 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.332917 7059 scope.go:122] "RemoveContainer" containerID="bc649ae8fe26ee698d8c31f856e194fd2da184ae32004b377d05bc5cf06bd468" Nov 04 01:35:53.334530 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.334498 7059 generic.go:356] "Generic (PLEG): container finished" podID="507f1fa6-dc41-4285-9e97-941eb8e9f525" containerID="b53b2c978346dd9a897ea0aa2ac729d2e5a0924eff3c68090e0b573a5cd6e554" exitCode=0 Nov 04 01:35:53.334610 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.334553 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d7e45d-7ff9858d99-qv8gj" Nov 04 01:35:53.334676 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.334531 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d7e45d-7ff9858d99-qv8gj" event={"ID":"507f1fa6-dc41-4285-9e97-941eb8e9f525","Type":"ContainerDied","Data":"b53b2c978346dd9a897ea0aa2ac729d2e5a0924eff3c68090e0b573a5cd6e554"} Nov 04 01:35:53.334725 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.334692 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d7e45d-7ff9858d99-qv8gj" event={"ID":"507f1fa6-dc41-4285-9e97-941eb8e9f525","Type":"ContainerDied","Data":"5285cea581c50ef2e87c4767c1f3f66d5b736baf4a1105589e8c7dae0760f8cb"} Nov 04 01:35:53.335136 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.335105 7059 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" secret="" err="secret \"e2e-worker-w6-499ddd-openshift-default-dockercfg-fb8nq\" not found" Nov 04 01:35:53.343784 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.343762 7059 scope.go:122] "RemoveContainer" containerID="bc649ae8fe26ee698d8c31f856e194fd2da184ae32004b377d05bc5cf06bd468" Nov 04 01:35:53.344153 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:35:53.344135 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc649ae8fe26ee698d8c31f856e194fd2da184ae32004b377d05bc5cf06bd468\": container with ID starting with bc649ae8fe26ee698d8c31f856e194fd2da184ae32004b377d05bc5cf06bd468 not found: ID does not exist" containerID="bc649ae8fe26ee698d8c31f856e194fd2da184ae32004b377d05bc5cf06bd468" Nov 04 01:35:53.344211 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.344164 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc649ae8fe26ee698d8c31f856e194fd2da184ae32004b377d05bc5cf06bd468"} err="failed to get container status \"bc649ae8fe26ee698d8c31f856e194fd2da184ae32004b377d05bc5cf06bd468\": rpc error: code = NotFound desc = could not find container \"bc649ae8fe26ee698d8c31f856e194fd2da184ae32004b377d05bc5cf06bd468\": container with ID starting with bc649ae8fe26ee698d8c31f856e194fd2da184ae32004b377d05bc5cf06bd468 not found: ID does not exist" Nov 04 01:35:53.344211 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.344181 7059 scope.go:122] "RemoveContainer" containerID="b53b2c978346dd9a897ea0aa2ac729d2e5a0924eff3c68090e0b573a5cd6e554" Nov 04 01:35:53.354412 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.354391 7059 scope.go:122] "RemoveContainer" containerID="b53b2c978346dd9a897ea0aa2ac729d2e5a0924eff3c68090e0b573a5cd6e554" Nov 04 01:35:53.354695 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:35:53.354662 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b53b2c978346dd9a897ea0aa2ac729d2e5a0924eff3c68090e0b573a5cd6e554\": container with ID starting with b53b2c978346dd9a897ea0aa2ac729d2e5a0924eff3c68090e0b573a5cd6e554 not found: ID does not exist" containerID="b53b2c978346dd9a897ea0aa2ac729d2e5a0924eff3c68090e0b573a5cd6e554" Nov 04 01:35:53.354740 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.354702 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b53b2c978346dd9a897ea0aa2ac729d2e5a0924eff3c68090e0b573a5cd6e554"} err="failed to get container status \"b53b2c978346dd9a897ea0aa2ac729d2e5a0924eff3c68090e0b573a5cd6e554\": rpc error: code = NotFound desc = could not find container \"b53b2c978346dd9a897ea0aa2ac729d2e5a0924eff3c68090e0b573a5cd6e554\": container with ID starting with b53b2c978346dd9a897ea0aa2ac729d2e5a0924eff3c68090e0b573a5cd6e554 not found: ID does not exist" Nov 04 01:35:53.363655 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.363625 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d7e45d-7ff9858d99-qv8gj"] Nov 04 01:35:53.373169 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.373144 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d7e45d-7ff9858d99-qv8gj"] Nov 04 01:35:53.385332 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.385301 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d7e45d-78c9c88f96-sggpl"] Nov 04 01:35:53.387962 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:53.387936 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d7e45d-78c9c88f96-sggpl"] Nov 04 01:35:54.342468 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:54.342429 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" podUID="44f12c1b-9d91-4575-973e-cd1aadebf87c" containerName="istio-proxy" containerID="cri-o://85598e395bc0caae1a06fc5c10b62b5c1160ebc1170fcd738ca8823169634dcc" gracePeriod=30 Nov 04 01:35:55.176390 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:55.176355 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="507f1fa6-dc41-4285-9e97-941eb8e9f525" path="/var/lib/kubelet/pods/507f1fa6-dc41-4285-9e97-941eb8e9f525/volumes" Nov 04 01:35:55.176743 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:55.176729 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d090174e-7ec3-4c2c-a296-f15f9543ea36" path="/var/lib/kubelet/pods/d090174e-7ec3-4c2c-a296-f15f9543ea36/volumes" Nov 04 01:35:58.579638 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.579583 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:35:58.598995 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.598972 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:35:58.727975 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.727940 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b925870-1345-4a77-8f3b-2175eb9bb93a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b925870-1345-4a77-8f3b-2175eb9bb93a-istio-podinfo\") pod \"5b925870-1345-4a77-8f3b-2175eb9bb93a\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " Nov 04 01:35:58.728173 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.728010 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-istio-envoy\") pod \"5b925870-1345-4a77-8f3b-2175eb9bb93a\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " Nov 04 01:35:58.728173 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.728053 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istio-podinfo\") pod \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " Nov 04 01:35:58.728173 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.728081 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istio-envoy\") pod \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " Nov 04 01:35:58.728173 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.728140 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b925870-1345-4a77-8f3b-2175eb9bb93a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b925870-1345-4a77-8f3b-2175eb9bb93a-istiod-ca-cert\") pod \"5b925870-1345-4a77-8f3b-2175eb9bb93a\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " Nov 04 01:35:58.728337 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.728189 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-credential-socket\") pod \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " Nov 04 01:35:58.728337 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.728224 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istio-data\") pod \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " Nov 04 01:35:58.728337 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.728248 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-workload-certs\") pod \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " Nov 04 01:35:58.728337 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.728275 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-istio-data\") pod \"5b925870-1345-4a77-8f3b-2175eb9bb93a\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " Nov 04 01:35:58.728337 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.728302 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-credential-socket\") pod \"5b925870-1345-4a77-8f3b-2175eb9bb93a\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " Nov 04 01:35:58.728337 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.728323 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b925870-1345-4a77-8f3b-2175eb9bb93a-kube-api-access-sh92h\" (UniqueName: \"kubernetes.io/projected/5b925870-1345-4a77-8f3b-2175eb9bb93a-kube-api-access-sh92h\") pod \"5b925870-1345-4a77-8f3b-2175eb9bb93a\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " Nov 04 01:35:58.728636 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.728353 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-workload-certs\") pod \"5b925870-1345-4a77-8f3b-2175eb9bb93a\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " Nov 04 01:35:58.728636 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.728370 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b925870-1345-4a77-8f3b-2175eb9bb93a-istio-token\" (UniqueName: \"kubernetes.io/projected/5b925870-1345-4a77-8f3b-2175eb9bb93a-istio-token\") pod \"5b925870-1345-4a77-8f3b-2175eb9bb93a\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " Nov 04 01:35:58.728636 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.728395 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istio-token\" (UniqueName: \"kubernetes.io/projected/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istio-token\") pod \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " Nov 04 01:35:58.728636 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.728426 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-workload-socket\") pod \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " Nov 04 01:35:58.728636 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.728497 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-kube-api-access-qznkf\" (UniqueName: \"kubernetes.io/projected/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-kube-api-access-qznkf\") pod \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " Nov 04 01:35:58.728636 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.728529 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istiod-ca-cert\") pod \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\" (UID: \"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b\") " Nov 04 01:35:58.728636 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.728575 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-workload-socket\") pod \"5b925870-1345-4a77-8f3b-2175eb9bb93a\" (UID: \"5b925870-1345-4a77-8f3b-2175eb9bb93a\") " Nov 04 01:35:58.729490 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.729286 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-istio-data" pod "5b925870-1345-4a77-8f3b-2175eb9bb93a" (UID: "5b925870-1345-4a77-8f3b-2175eb9bb93a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:58.729490 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.729294 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-workload-socket" pod "5b925870-1345-4a77-8f3b-2175eb9bb93a" (UID: "5b925870-1345-4a77-8f3b-2175eb9bb93a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:58.729490 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.729441 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-credential-socket" pod "5b925870-1345-4a77-8f3b-2175eb9bb93a" (UID: "5b925870-1345-4a77-8f3b-2175eb9bb93a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:58.729716 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.729548 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-workload-socket" pod "bf384bd4-f417-42a3-b8a2-cddcbddc8d8b" (UID: "bf384bd4-f417-42a3-b8a2-cddcbddc8d8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:58.729716 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.729660 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istiod-ca-cert" pod "bf384bd4-f417-42a3-b8a2-cddcbddc8d8b" (UID: "bf384bd4-f417-42a3-b8a2-cddcbddc8d8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:58.730822 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.729899 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-workload-certs" pod "5b925870-1345-4a77-8f3b-2175eb9bb93a" (UID: "5b925870-1345-4a77-8f3b-2175eb9bb93a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:58.730822 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.729961 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istio-data" pod "bf384bd4-f417-42a3-b8a2-cddcbddc8d8b" (UID: "bf384bd4-f417-42a3-b8a2-cddcbddc8d8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:58.730822 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.730323 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-credential-socket" pod "bf384bd4-f417-42a3-b8a2-cddcbddc8d8b" (UID: "bf384bd4-f417-42a3-b8a2-cddcbddc8d8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:58.730822 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.730509 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-workload-certs" pod "bf384bd4-f417-42a3-b8a2-cddcbddc8d8b" (UID: "bf384bd4-f417-42a3-b8a2-cddcbddc8d8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:58.730822 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.730543 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b925870-1345-4a77-8f3b-2175eb9bb93a-istiod-ca-cert" pod "5b925870-1345-4a77-8f3b-2175eb9bb93a" (UID: "5b925870-1345-4a77-8f3b-2175eb9bb93a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:58.732149 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.732120 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istio-token" pod "bf384bd4-f417-42a3-b8a2-cddcbddc8d8b" (UID: "bf384bd4-f417-42a3-b8a2-cddcbddc8d8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:58.732261 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.732203 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-kube-api-access-qznkf" pod "bf384bd4-f417-42a3-b8a2-cddcbddc8d8b" (UID: "bf384bd4-f417-42a3-b8a2-cddcbddc8d8b"). InnerVolumeSpecName "kube-api-access-qznkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:58.732261 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.732220 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b925870-1345-4a77-8f3b-2175eb9bb93a-istio-podinfo" pod "5b925870-1345-4a77-8f3b-2175eb9bb93a" (UID: "5b925870-1345-4a77-8f3b-2175eb9bb93a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:35:58.732707 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.732685 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istio-envoy" pod "bf384bd4-f417-42a3-b8a2-cddcbddc8d8b" (UID: "bf384bd4-f417-42a3-b8a2-cddcbddc8d8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:58.732846 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.732830 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b925870-1345-4a77-8f3b-2175eb9bb93a-kube-api-access-sh92h" pod "5b925870-1345-4a77-8f3b-2175eb9bb93a" (UID: "5b925870-1345-4a77-8f3b-2175eb9bb93a"). InnerVolumeSpecName "kube-api-access-sh92h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:58.734330 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.734305 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istio-podinfo" pod "bf384bd4-f417-42a3-b8a2-cddcbddc8d8b" (UID: "bf384bd4-f417-42a3-b8a2-cddcbddc8d8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:35:58.734433 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.734411 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-istio-envoy" pod "5b925870-1345-4a77-8f3b-2175eb9bb93a" (UID: "5b925870-1345-4a77-8f3b-2175eb9bb93a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:58.734516 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.734499 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b925870-1345-4a77-8f3b-2175eb9bb93a-istio-token" pod "5b925870-1345-4a77-8f3b-2175eb9bb93a" (UID: "5b925870-1345-4a77-8f3b-2175eb9bb93a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:58.830539 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.830466 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.830539 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.830492 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.830539 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.830503 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.830539 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.830511 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b925870-1345-4a77-8f3b-2175eb9bb93a-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.830539 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.830520 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.830539 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.830528 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.830539 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.830536 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.830539 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.830544 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.831100 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.830552 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.831100 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.830561 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh92h\" (UniqueName: \"kubernetes.io/projected/5b925870-1345-4a77-8f3b-2175eb9bb93a-kube-api-access-sh92h\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.831100 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.830570 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.831100 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.830578 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b925870-1345-4a77-8f3b-2175eb9bb93a-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.831100 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.830586 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.831100 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.830594 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.831100 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.830603 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qznkf\" (UniqueName: \"kubernetes.io/projected/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-kube-api-access-qznkf\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.831100 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.830611 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.831100 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.830620 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b925870-1345-4a77-8f3b-2175eb9bb93a-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.831100 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:58.830628 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b925870-1345-4a77-8f3b-2175eb9bb93a-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:59.373415 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.373387 7059 generic.go:356] "Generic (PLEG): container finished" podID="5b925870-1345-4a77-8f3b-2175eb9bb93a" containerID="8da6d1bb6c9da5ab033ee837b326722c8fb4bca4bf7b5c0bf79cc1cce8fa7bda" exitCode=0 Nov 04 01:35:59.373538 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.373434 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" Nov 04 01:35:59.373538 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.373460 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" event={"ID":"5b925870-1345-4a77-8f3b-2175eb9bb93a","Type":"ContainerDied","Data":"8da6d1bb6c9da5ab033ee837b326722c8fb4bca4bf7b5c0bf79cc1cce8fa7bda"} Nov 04 01:35:59.373538 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.373509 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw" event={"ID":"5b925870-1345-4a77-8f3b-2175eb9bb93a","Type":"ContainerDied","Data":"39cb9c650b30eb1deda9be28f0a64b1c0712837dcff7c8ce962f700b3088827b"} Nov 04 01:35:59.373538 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.373534 7059 scope.go:122] "RemoveContainer" containerID="8da6d1bb6c9da5ab033ee837b326722c8fb4bca4bf7b5c0bf79cc1cce8fa7bda" Nov 04 01:35:59.374887 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.374867 7059 generic.go:356] "Generic (PLEG): container finished" podID="bf384bd4-f417-42a3-b8a2-cddcbddc8d8b" containerID="489d393bc645ccd2293f47d7ecd79b0f7743b38ae03b8d48cc61b4e08eb924a8" exitCode=0 Nov 04 01:35:59.375014 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.374902 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" Nov 04 01:35:59.375014 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.374936 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" event={"ID":"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b","Type":"ContainerDied","Data":"489d393bc645ccd2293f47d7ecd79b0f7743b38ae03b8d48cc61b4e08eb924a8"} Nov 04 01:35:59.375014 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.374978 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq" event={"ID":"bf384bd4-f417-42a3-b8a2-cddcbddc8d8b","Type":"ContainerDied","Data":"72ad38a368c6f707a8e9390f62dbc5108e7cede70cfcbada7775f6f7f628465a"} Nov 04 01:35:59.428165 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.428138 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw"] Nov 04 01:35:59.430481 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.430462 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5wbcw"] Nov 04 01:35:59.439461 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.439416 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq"] Nov 04 01:35:59.444935 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.444906 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-5mjsq"] Nov 04 01:35:59.454985 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.454852 7059 scope.go:122] "RemoveContainer" containerID="8da6d1bb6c9da5ab033ee837b326722c8fb4bca4bf7b5c0bf79cc1cce8fa7bda" Nov 04 01:35:59.455193 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:35:59.455172 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8da6d1bb6c9da5ab033ee837b326722c8fb4bca4bf7b5c0bf79cc1cce8fa7bda\": container with ID starting with 8da6d1bb6c9da5ab033ee837b326722c8fb4bca4bf7b5c0bf79cc1cce8fa7bda not found: ID does not exist" containerID="8da6d1bb6c9da5ab033ee837b326722c8fb4bca4bf7b5c0bf79cc1cce8fa7bda" Nov 04 01:35:59.455270 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.455205 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8da6d1bb6c9da5ab033ee837b326722c8fb4bca4bf7b5c0bf79cc1cce8fa7bda"} err="failed to get container status \"8da6d1bb6c9da5ab033ee837b326722c8fb4bca4bf7b5c0bf79cc1cce8fa7bda\": rpc error: code = NotFound desc = could not find container \"8da6d1bb6c9da5ab033ee837b326722c8fb4bca4bf7b5c0bf79cc1cce8fa7bda\": container with ID starting with 8da6d1bb6c9da5ab033ee837b326722c8fb4bca4bf7b5c0bf79cc1cce8fa7bda not found: ID does not exist" Nov 04 01:35:59.455270 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.455229 7059 scope.go:122] "RemoveContainer" containerID="489d393bc645ccd2293f47d7ecd79b0f7743b38ae03b8d48cc61b4e08eb924a8" Nov 04 01:35:59.463735 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.463720 7059 scope.go:122] "RemoveContainer" containerID="489d393bc645ccd2293f47d7ecd79b0f7743b38ae03b8d48cc61b4e08eb924a8" Nov 04 01:35:59.464070 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:35:59.464047 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"489d393bc645ccd2293f47d7ecd79b0f7743b38ae03b8d48cc61b4e08eb924a8\": container with ID starting with 489d393bc645ccd2293f47d7ecd79b0f7743b38ae03b8d48cc61b4e08eb924a8 not found: ID does not exist" containerID="489d393bc645ccd2293f47d7ecd79b0f7743b38ae03b8d48cc61b4e08eb924a8" Nov 04 01:35:59.464171 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.464075 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"489d393bc645ccd2293f47d7ecd79b0f7743b38ae03b8d48cc61b4e08eb924a8"} err="failed to get container status \"489d393bc645ccd2293f47d7ecd79b0f7743b38ae03b8d48cc61b4e08eb924a8\": rpc error: code = NotFound desc = could not find container \"489d393bc645ccd2293f47d7ecd79b0f7743b38ae03b8d48cc61b4e08eb924a8\": container with ID starting with 489d393bc645ccd2293f47d7ecd79b0f7743b38ae03b8d48cc61b4e08eb924a8 not found: ID does not exist" Nov 04 01:35:59.597515 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.597486 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:35:59.739087 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.739006 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44f12c1b-9d91-4575-973e-cd1aadebf87c-istio-token\" (UniqueName: \"kubernetes.io/projected/44f12c1b-9d91-4575-973e-cd1aadebf87c-istio-token\") pod \"44f12c1b-9d91-4575-973e-cd1aadebf87c\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " Nov 04 01:35:59.739231 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.739092 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-istio-data\") pod \"44f12c1b-9d91-4575-973e-cd1aadebf87c\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " Nov 04 01:35:59.739231 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.739151 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44f12c1b-9d91-4575-973e-cd1aadebf87c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f12c1b-9d91-4575-973e-cd1aadebf87c-istio-podinfo\") pod \"44f12c1b-9d91-4575-973e-cd1aadebf87c\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " Nov 04 01:35:59.739231 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.739218 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-istio-envoy\") pod \"44f12c1b-9d91-4575-973e-cd1aadebf87c\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " Nov 04 01:35:59.739410 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.739248 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-credential-socket\") pod \"44f12c1b-9d91-4575-973e-cd1aadebf87c\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " Nov 04 01:35:59.739410 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.739295 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44f12c1b-9d91-4575-973e-cd1aadebf87c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f12c1b-9d91-4575-973e-cd1aadebf87c-istiod-ca-cert\") pod \"44f12c1b-9d91-4575-973e-cd1aadebf87c\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " Nov 04 01:35:59.739410 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.739327 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44f12c1b-9d91-4575-973e-cd1aadebf87c-kube-api-access-cwgr8\" (UniqueName: \"kubernetes.io/projected/44f12c1b-9d91-4575-973e-cd1aadebf87c-kube-api-access-cwgr8\") pod \"44f12c1b-9d91-4575-973e-cd1aadebf87c\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " Nov 04 01:35:59.739410 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.739399 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-istio-data" pod "44f12c1b-9d91-4575-973e-cd1aadebf87c" (UID: "44f12c1b-9d91-4575-973e-cd1aadebf87c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:59.739639 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.739404 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-workload-socket\") pod \"44f12c1b-9d91-4575-973e-cd1aadebf87c\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " Nov 04 01:35:59.739639 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.739555 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-workload-certs\") pod \"44f12c1b-9d91-4575-973e-cd1aadebf87c\" (UID: \"44f12c1b-9d91-4575-973e-cd1aadebf87c\") " Nov 04 01:35:59.739639 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.739542 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-credential-socket" pod "44f12c1b-9d91-4575-973e-cd1aadebf87c" (UID: "44f12c1b-9d91-4575-973e-cd1aadebf87c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:59.739639 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.739611 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-workload-socket" pod "44f12c1b-9d91-4575-973e-cd1aadebf87c" (UID: "44f12c1b-9d91-4575-973e-cd1aadebf87c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:59.739945 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.739840 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-workload-certs" pod "44f12c1b-9d91-4575-973e-cd1aadebf87c" (UID: "44f12c1b-9d91-4575-973e-cd1aadebf87c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:59.740139 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.740011 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44f12c1b-9d91-4575-973e-cd1aadebf87c-istiod-ca-cert" pod "44f12c1b-9d91-4575-973e-cd1aadebf87c" (UID: "44f12c1b-9d91-4575-973e-cd1aadebf87c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:59.740139 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.740039 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:59.740139 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.740051 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:59.740139 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.740060 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:59.740139 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.740069 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:59.741439 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.741412 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f12c1b-9d91-4575-973e-cd1aadebf87c-istio-token" pod "44f12c1b-9d91-4575-973e-cd1aadebf87c" (UID: "44f12c1b-9d91-4575-973e-cd1aadebf87c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:59.741536 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.741505 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44f12c1b-9d91-4575-973e-cd1aadebf87c-istio-podinfo" pod "44f12c1b-9d91-4575-973e-cd1aadebf87c" (UID: "44f12c1b-9d91-4575-973e-cd1aadebf87c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:35:59.741726 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.741706 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f12c1b-9d91-4575-973e-cd1aadebf87c-kube-api-access-cwgr8" pod "44f12c1b-9d91-4575-973e-cd1aadebf87c" (UID: "44f12c1b-9d91-4575-973e-cd1aadebf87c"). InnerVolumeSpecName "kube-api-access-cwgr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:59.741817 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.741801 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-istio-envoy" pod "44f12c1b-9d91-4575-973e-cd1aadebf87c" (UID: "44f12c1b-9d91-4575-973e-cd1aadebf87c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:59.840786 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.840760 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44f12c1b-9d91-4575-973e-cd1aadebf87c-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:59.840786 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.840784 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44f12c1b-9d91-4575-973e-cd1aadebf87c-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:59.840969 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.840794 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44f12c1b-9d91-4575-973e-cd1aadebf87c-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:59.840969 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.840802 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44f12c1b-9d91-4575-973e-cd1aadebf87c-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:59.840969 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:35:59.840810 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwgr8\" (UniqueName: \"kubernetes.io/projected/44f12c1b-9d91-4575-973e-cd1aadebf87c-kube-api-access-cwgr8\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:00.382149 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:00.382104 7059 generic.go:356] "Generic (PLEG): container finished" podID="44f12c1b-9d91-4575-973e-cd1aadebf87c" containerID="85598e395bc0caae1a06fc5c10b62b5c1160ebc1170fcd738ca8823169634dcc" exitCode=0 Nov 04 01:36:00.382340 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:00.382174 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" Nov 04 01:36:00.382340 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:00.382188 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" event={"ID":"44f12c1b-9d91-4575-973e-cd1aadebf87c","Type":"ContainerDied","Data":"85598e395bc0caae1a06fc5c10b62b5c1160ebc1170fcd738ca8823169634dcc"} Nov 04 01:36:00.382340 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:00.382252 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg" event={"ID":"44f12c1b-9d91-4575-973e-cd1aadebf87c","Type":"ContainerDied","Data":"6af3eef21a13fe188ef5a3d423f366a4969e09ad5fbb6cc7bb8b0c7fe8ba98b4"} Nov 04 01:36:00.382340 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:00.382282 7059 scope.go:122] "RemoveContainer" containerID="85598e395bc0caae1a06fc5c10b62b5c1160ebc1170fcd738ca8823169634dcc" Nov 04 01:36:00.404327 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:00.394783 7059 scope.go:122] "RemoveContainer" containerID="85598e395bc0caae1a06fc5c10b62b5c1160ebc1170fcd738ca8823169634dcc" Nov 04 01:36:00.404327 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:36:00.395201 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85598e395bc0caae1a06fc5c10b62b5c1160ebc1170fcd738ca8823169634dcc\": container with ID starting with 85598e395bc0caae1a06fc5c10b62b5c1160ebc1170fcd738ca8823169634dcc not found: ID does not exist" containerID="85598e395bc0caae1a06fc5c10b62b5c1160ebc1170fcd738ca8823169634dcc" Nov 04 01:36:00.404327 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:00.395239 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85598e395bc0caae1a06fc5c10b62b5c1160ebc1170fcd738ca8823169634dcc"} err="failed to get container status \"85598e395bc0caae1a06fc5c10b62b5c1160ebc1170fcd738ca8823169634dcc\": rpc error: code = NotFound desc = could not find container \"85598e395bc0caae1a06fc5c10b62b5c1160ebc1170fcd738ca8823169634dcc\": container with ID starting with 85598e395bc0caae1a06fc5c10b62b5c1160ebc1170fcd738ca8823169634dcc not found: ID does not exist" Nov 04 01:36:00.461568 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:00.444284 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg"] Nov 04 01:36:00.465595 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:00.465562 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nzjwg"] Nov 04 01:36:00.466422 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:00.466399 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499"] Nov 04 01:36:00.466653 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:00.466620 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" podUID="b42e3b72-1c9f-4a14-88c8-755b3faceb5a" containerName="istio-proxy" containerID="cri-o://090fb0a7cd2fd554cac2320d82f5c4dda1a6fe7de2c781dcf90517fb71d38819" gracePeriod=30 Nov 04 01:36:00.470054 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:00.470032 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7"] Nov 04 01:36:00.470238 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:00.470216 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" podUID="7e59c4bd-a00f-41e1-8d1c-610b9d791374" containerName="istio-proxy" containerID="cri-o://7f68e8df09a677e526e8ba13b3b5bd8a1e9949fa96226e762581834255b5d0ff" gracePeriod=30 Nov 04 01:36:00.472834 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:00.472810 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9"] Nov 04 01:36:00.473103 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:00.473078 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" podUID="eedbad81-5540-45d7-adbb-54afd75aa6e8" containerName="istio-proxy" containerID="cri-o://a3d4097794e797d260b9122d55862b113ff5987deb7606389f7dd8d59ce065d1" gracePeriod=30 Nov 04 01:36:01.177381 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:01.177336 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44f12c1b-9d91-4575-973e-cd1aadebf87c" path="/var/lib/kubelet/pods/44f12c1b-9d91-4575-973e-cd1aadebf87c/volumes" Nov 04 01:36:01.177988 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:01.177968 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b925870-1345-4a77-8f3b-2175eb9bb93a" path="/var/lib/kubelet/pods/5b925870-1345-4a77-8f3b-2175eb9bb93a/volumes" Nov 04 01:36:01.178436 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:01.178417 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf384bd4-f417-42a3-b8a2-cddcbddc8d8b" path="/var/lib/kubelet/pods/bf384bd4-f417-42a3-b8a2-cddcbddc8d8b/volumes" Nov 04 01:36:03.161177 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:03.161143 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8498f949d-mgqrt"] Nov 04 01:36:03.161718 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:03.161492 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-8498f949d-mgqrt" podUID="69707113-e314-4ee5-9e00-0fd663820d62" containerName="manager" containerID="cri-o://a0c4df770db604bb9c4d793e81877c4fa039e35623300fc2d04abb3b8a843267" gracePeriod=30 Nov 04 01:36:03.405267 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:03.405233 7059 generic.go:356] "Generic (PLEG): container finished" podID="69707113-e314-4ee5-9e00-0fd663820d62" containerID="a0c4df770db604bb9c4d793e81877c4fa039e35623300fc2d04abb3b8a843267" exitCode=0 Nov 04 01:36:03.405399 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:03.405298 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8498f949d-mgqrt" event={"ID":"69707113-e314-4ee5-9e00-0fd663820d62","Type":"ContainerDied","Data":"a0c4df770db604bb9c4d793e81877c4fa039e35623300fc2d04abb3b8a843267"} Nov 04 01:36:03.423869 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:03.423853 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8498f949d-mgqrt" Nov 04 01:36:03.594477 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:03.594445 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69707113-e314-4ee5-9e00-0fd663820d62-webhook-cert\" (UniqueName: \"kubernetes.io/secret/69707113-e314-4ee5-9e00-0fd663820d62-webhook-cert\") pod \"69707113-e314-4ee5-9e00-0fd663820d62\" (UID: \"69707113-e314-4ee5-9e00-0fd663820d62\") " Nov 04 01:36:03.594624 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:03.594511 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69707113-e314-4ee5-9e00-0fd663820d62-kube-api-access-s2nc9\" (UniqueName: \"kubernetes.io/projected/69707113-e314-4ee5-9e00-0fd663820d62-kube-api-access-s2nc9\") pod \"69707113-e314-4ee5-9e00-0fd663820d62\" (UID: \"69707113-e314-4ee5-9e00-0fd663820d62\") " Nov 04 01:36:03.594624 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:03.594605 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69707113-e314-4ee5-9e00-0fd663820d62-metrics-certs\" (UniqueName: \"kubernetes.io/secret/69707113-e314-4ee5-9e00-0fd663820d62-metrics-certs\") pod \"69707113-e314-4ee5-9e00-0fd663820d62\" (UID: \"69707113-e314-4ee5-9e00-0fd663820d62\") " Nov 04 01:36:03.596854 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:03.596828 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69707113-e314-4ee5-9e00-0fd663820d62-metrics-certs" pod "69707113-e314-4ee5-9e00-0fd663820d62" (UID: "69707113-e314-4ee5-9e00-0fd663820d62"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:36:03.597000 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:03.596871 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69707113-e314-4ee5-9e00-0fd663820d62-kube-api-access-s2nc9" pod "69707113-e314-4ee5-9e00-0fd663820d62" (UID: "69707113-e314-4ee5-9e00-0fd663820d62"). InnerVolumeSpecName "kube-api-access-s2nc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:03.597000 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:03.596874 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69707113-e314-4ee5-9e00-0fd663820d62-webhook-cert" pod "69707113-e314-4ee5-9e00-0fd663820d62" (UID: "69707113-e314-4ee5-9e00-0fd663820d62"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:36:03.696477 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:03.696421 7059 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69707113-e314-4ee5-9e00-0fd663820d62-webhook-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:03.696477 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:03.696446 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2nc9\" (UniqueName: \"kubernetes.io/projected/69707113-e314-4ee5-9e00-0fd663820d62-kube-api-access-s2nc9\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:03.696477 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:03.696456 7059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69707113-e314-4ee5-9e00-0fd663820d62-metrics-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:04.412071 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:04.412044 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8498f949d-mgqrt" Nov 04 01:36:04.412574 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:04.412084 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8498f949d-mgqrt" event={"ID":"69707113-e314-4ee5-9e00-0fd663820d62","Type":"ContainerDied","Data":"10842f4b7de158085dd815fac1cc3f15ddf77530777608e12bf13e4a7919ce76"} Nov 04 01:36:04.412574 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:04.412145 7059 scope.go:122] "RemoveContainer" containerID="a0c4df770db604bb9c4d793e81877c4fa039e35623300fc2d04abb3b8a843267" Nov 04 01:36:04.435298 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:04.435265 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8498f949d-mgqrt"] Nov 04 01:36:04.440494 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:04.440472 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8498f949d-mgqrt"] Nov 04 01:36:04.911764 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:04.911720 7059 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:36:04.911959 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:04.911783 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" podUID="7e59c4bd-a00f-41e1-8d1c-610b9d791374" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:36:05.168779 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.168703 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8498f949d-mpjs7"] Nov 04 01:36:05.169297 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.169278 7059 cpu_manager.go:414] "containerMap: removing container" podUID="5b925870-1345-4a77-8f3b-2175eb9bb93a" containerName="istio-proxy" Nov 04 01:36:05.169381 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.169299 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b925870-1345-4a77-8f3b-2175eb9bb93a" containerName="istio-proxy" Nov 04 01:36:05.169381 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.169331 7059 cpu_manager.go:414] "containerMap: removing container" podUID="69707113-e314-4ee5-9e00-0fd663820d62" containerName="manager" Nov 04 01:36:05.169381 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.169340 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69707113-e314-4ee5-9e00-0fd663820d62" containerName="manager" Nov 04 01:36:05.169381 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.169355 7059 cpu_manager.go:414] "containerMap: removing container" podUID="22a00871-73e9-4c9a-aaa6-9046b1004b9e" containerName="payload-processing" Nov 04 01:36:05.169381 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.169362 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22a00871-73e9-4c9a-aaa6-9046b1004b9e" containerName="payload-processing" Nov 04 01:36:05.169381 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.169375 7059 cpu_manager.go:414] "containerMap: removing container" podUID="bf384bd4-f417-42a3-b8a2-cddcbddc8d8b" containerName="istio-proxy" Nov 04 01:36:05.169705 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.169383 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf384bd4-f417-42a3-b8a2-cddcbddc8d8b" containerName="istio-proxy" Nov 04 01:36:05.169705 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.169403 7059 cpu_manager.go:414] "containerMap: removing container" podUID="1510cc32-e22b-46f0-acfb-3e2994767889" containerName="payload-pre-processing" Nov 04 01:36:05.169705 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.169411 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1510cc32-e22b-46f0-acfb-3e2994767889" containerName="payload-pre-processing" Nov 04 01:36:05.169705 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.169430 7059 cpu_manager.go:414] "containerMap: removing container" podUID="44f12c1b-9d91-4575-973e-cd1aadebf87c" containerName="istio-proxy" Nov 04 01:36:05.169705 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.169438 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44f12c1b-9d91-4575-973e-cd1aadebf87c" containerName="istio-proxy" Nov 04 01:36:05.169705 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.169465 7059 cpu_manager.go:414] "containerMap: removing container" podUID="507f1fa6-dc41-4285-9e97-941eb8e9f525" containerName="payload-pre-processing" Nov 04 01:36:05.169705 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.169472 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="507f1fa6-dc41-4285-9e97-941eb8e9f525" containerName="payload-pre-processing" Nov 04 01:36:05.169705 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.169483 7059 cpu_manager.go:414] "containerMap: removing container" podUID="d090174e-7ec3-4c2c-a296-f15f9543ea36" containerName="payload-processing" Nov 04 01:36:05.169705 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.169490 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d090174e-7ec3-4c2c-a296-f15f9543ea36" containerName="payload-processing" Nov 04 01:36:05.169705 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.169587 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1510cc32-e22b-46f0-acfb-3e2994767889" containerName="payload-pre-processing" Nov 04 01:36:05.169705 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.169601 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="44f12c1b-9d91-4575-973e-cd1aadebf87c" containerName="istio-proxy" Nov 04 01:36:05.169705 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.169615 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="507f1fa6-dc41-4285-9e97-941eb8e9f525" containerName="payload-pre-processing" Nov 04 01:36:05.169705 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.169623 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d090174e-7ec3-4c2c-a296-f15f9543ea36" containerName="payload-processing" Nov 04 01:36:05.169705 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.169636 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b925870-1345-4a77-8f3b-2175eb9bb93a" containerName="istio-proxy" Nov 04 01:36:05.169705 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.169647 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="69707113-e314-4ee5-9e00-0fd663820d62" containerName="manager" Nov 04 01:36:05.169705 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.169657 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="22a00871-73e9-4c9a-aaa6-9046b1004b9e" containerName="payload-processing" Nov 04 01:36:05.169705 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.169666 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf384bd4-f417-42a3-b8a2-cddcbddc8d8b" containerName="istio-proxy" Nov 04 01:36:05.175701 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.175662 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8498f949d-mpjs7" Nov 04 01:36:05.181945 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.181285 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 04 01:36:05.185951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.184195 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 04 01:36:05.185951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.184280 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 04 01:36:05.190782 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.190755 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69707113-e314-4ee5-9e00-0fd663820d62" path="/var/lib/kubelet/pods/69707113-e314-4ee5-9e00-0fd663820d62/volumes" Nov 04 01:36:05.191806 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.191782 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8498f949d-mpjs7"] Nov 04 01:36:05.279825 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.279794 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nlglx\"" Nov 04 01:36:05.318433 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.318405 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8816836c-a119-4597-9909-778308ea72bc-metrics-certs\") pod \"maas-controller-8498f949d-mpjs7\" (UID: \"8816836c-a119-4597-9909-778308ea72bc\") " pod="opendatahub/maas-controller-8498f949d-mpjs7" Nov 04 01:36:05.318588 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.318465 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8816836c-a119-4597-9909-778308ea72bc-webhook-cert\") pod \"maas-controller-8498f949d-mpjs7\" (UID: \"8816836c-a119-4597-9909-778308ea72bc\") " pod="opendatahub/maas-controller-8498f949d-mpjs7" Nov 04 01:36:05.318588 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.318491 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46hwj\" (UniqueName: \"kubernetes.io/projected/8816836c-a119-4597-9909-778308ea72bc-kube-api-access-46hwj\") pod \"maas-controller-8498f949d-mpjs7\" (UID: \"8816836c-a119-4597-9909-778308ea72bc\") " pod="opendatahub/maas-controller-8498f949d-mpjs7" Nov 04 01:36:05.419902 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.419806 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8816836c-a119-4597-9909-778308ea72bc-metrics-certs\") pod \"maas-controller-8498f949d-mpjs7\" (UID: \"8816836c-a119-4597-9909-778308ea72bc\") " pod="opendatahub/maas-controller-8498f949d-mpjs7" Nov 04 01:36:05.419902 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.419883 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8816836c-a119-4597-9909-778308ea72bc-webhook-cert\") pod \"maas-controller-8498f949d-mpjs7\" (UID: \"8816836c-a119-4597-9909-778308ea72bc\") " pod="opendatahub/maas-controller-8498f949d-mpjs7" Nov 04 01:36:05.420447 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.419913 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46hwj\" (UniqueName: \"kubernetes.io/projected/8816836c-a119-4597-9909-778308ea72bc-kube-api-access-46hwj\") pod \"maas-controller-8498f949d-mpjs7\" (UID: \"8816836c-a119-4597-9909-778308ea72bc\") " pod="opendatahub/maas-controller-8498f949d-mpjs7" Nov 04 01:36:05.422896 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.422867 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8816836c-a119-4597-9909-778308ea72bc-webhook-cert\") pod \"maas-controller-8498f949d-mpjs7\" (UID: \"8816836c-a119-4597-9909-778308ea72bc\") " pod="opendatahub/maas-controller-8498f949d-mpjs7" Nov 04 01:36:05.423156 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.423130 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8816836c-a119-4597-9909-778308ea72bc-metrics-certs\") pod \"maas-controller-8498f949d-mpjs7\" (UID: \"8816836c-a119-4597-9909-778308ea72bc\") " pod="opendatahub/maas-controller-8498f949d-mpjs7" Nov 04 01:36:05.428127 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.428102 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46hwj\" (UniqueName: \"kubernetes.io/projected/8816836c-a119-4597-9909-778308ea72bc-kube-api-access-46hwj\") pod \"maas-controller-8498f949d-mpjs7\" (UID: \"8816836c-a119-4597-9909-778308ea72bc\") " pod="opendatahub/maas-controller-8498f949d-mpjs7" Nov 04 01:36:05.506453 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.506422 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8498f949d-mpjs7" Nov 04 01:36:05.937811 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.902160 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8498f949d-mpjs7"] Nov 04 01:36:05.984956 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:36:05.982563 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8816836c_a119_4597_9909_778308ea72bc.slice/crio-522a5c039b2a9d37c7bc3cad0160f9ce50e3bc69ae563cda78dadfc577cbb93b WatchSource:0}: Error finding container 522a5c039b2a9d37c7bc3cad0160f9ce50e3bc69ae563cda78dadfc577cbb93b: Status 404 returned error can't find the container with id 522a5c039b2a9d37c7bc3cad0160f9ce50e3bc69ae563cda78dadfc577cbb93b Nov 04 01:36:06.010200 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:05.998506 7059 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:36:06.236362 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.236018 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:36:06.268265 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.245088 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:36:06.332954 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.332690 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istio-envoy\") pod \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " Nov 04 01:36:06.332954 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.332739 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istio-token\" (UniqueName: \"kubernetes.io/projected/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istio-token\") pod \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " Nov 04 01:36:06.332954 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.332793 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istiod-ca-cert\") pod \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " Nov 04 01:36:06.332954 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.332816 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-workload-socket\") pod \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " Nov 04 01:36:06.332954 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.332887 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istio-data\") pod \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " Nov 04 01:36:06.332954 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.332903 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e59c4bd-a00f-41e1-8d1c-610b9d791374-kube-api-access-fwlfd\" (UniqueName: \"kubernetes.io/projected/7e59c4bd-a00f-41e1-8d1c-610b9d791374-kube-api-access-fwlfd\") pod \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " Nov 04 01:36:06.332954 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.332965 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-credential-socket\") pod \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " Nov 04 01:36:06.332954 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.332997 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-workload-certs\") pod \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " Nov 04 01:36:06.333784 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.333041 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istio-podinfo\") pod \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\" (UID: \"7e59c4bd-a00f-41e1-8d1c-610b9d791374\") " Nov 04 01:36:06.333784 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.333289 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istiod-ca-cert" pod "7e59c4bd-a00f-41e1-8d1c-610b9d791374" (UID: "7e59c4bd-a00f-41e1-8d1c-610b9d791374"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:36:06.333784 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.333518 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.334312 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.334277 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-credential-socket" pod "7e59c4bd-a00f-41e1-8d1c-610b9d791374" (UID: "7e59c4bd-a00f-41e1-8d1c-610b9d791374"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.334526 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.334407 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istio-data" pod "7e59c4bd-a00f-41e1-8d1c-610b9d791374" (UID: "7e59c4bd-a00f-41e1-8d1c-610b9d791374"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.334694 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.334426 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-workload-certs" pod "7e59c4bd-a00f-41e1-8d1c-610b9d791374" (UID: "7e59c4bd-a00f-41e1-8d1c-610b9d791374"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.334799 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.334624 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-workload-socket" pod "7e59c4bd-a00f-41e1-8d1c-610b9d791374" (UID: "7e59c4bd-a00f-41e1-8d1c-610b9d791374"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.335974 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.335917 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e59c4bd-a00f-41e1-8d1c-610b9d791374-kube-api-access-fwlfd" pod "7e59c4bd-a00f-41e1-8d1c-610b9d791374" (UID: "7e59c4bd-a00f-41e1-8d1c-610b9d791374"). InnerVolumeSpecName "kube-api-access-fwlfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:06.336132 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.336107 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istio-envoy" pod "7e59c4bd-a00f-41e1-8d1c-610b9d791374" (UID: "7e59c4bd-a00f-41e1-8d1c-610b9d791374"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.336362 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.336339 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istio-podinfo" pod "7e59c4bd-a00f-41e1-8d1c-610b9d791374" (UID: "7e59c4bd-a00f-41e1-8d1c-610b9d791374"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:36:06.336784 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.336767 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istio-token" pod "7e59c4bd-a00f-41e1-8d1c-610b9d791374" (UID: "7e59c4bd-a00f-41e1-8d1c-610b9d791374"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:06.370226 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.370181 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:36:06.435952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.435608 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istiod-ca-cert\") pod \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " Nov 04 01:36:06.435952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.435690 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-kube-api-access-vm9qp\" (UniqueName: \"kubernetes.io/projected/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-kube-api-access-vm9qp\") pod \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " Nov 04 01:36:06.435952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.435743 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eedbad81-5540-45d7-adbb-54afd75aa6e8-kube-api-access-ht2ln\" (UniqueName: \"kubernetes.io/projected/eedbad81-5540-45d7-adbb-54afd75aa6e8-kube-api-access-ht2ln\") pod \"eedbad81-5540-45d7-adbb-54afd75aa6e8\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " Nov 04 01:36:06.435952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.435772 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-credential-socket\") pod \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " Nov 04 01:36:06.435952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.435801 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istio-podinfo\") pod \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " Nov 04 01:36:06.435952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.435864 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-workload-certs\") pod \"eedbad81-5540-45d7-adbb-54afd75aa6e8\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " Nov 04 01:36:06.435952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.435896 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istio-envoy\") pod \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " Nov 04 01:36:06.436729 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.435968 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eedbad81-5540-45d7-adbb-54afd75aa6e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eedbad81-5540-45d7-adbb-54afd75aa6e8-istio-podinfo\") pod \"eedbad81-5540-45d7-adbb-54afd75aa6e8\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " Nov 04 01:36:06.436729 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.436016 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eedbad81-5540-45d7-adbb-54afd75aa6e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eedbad81-5540-45d7-adbb-54afd75aa6e8-istiod-ca-cert\") pod \"eedbad81-5540-45d7-adbb-54afd75aa6e8\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " Nov 04 01:36:06.436729 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.436055 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istio-data\") pod \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " Nov 04 01:36:06.436729 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.436089 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eedbad81-5540-45d7-adbb-54afd75aa6e8-istio-token\" (UniqueName: \"kubernetes.io/projected/eedbad81-5540-45d7-adbb-54afd75aa6e8-istio-token\") pod \"eedbad81-5540-45d7-adbb-54afd75aa6e8\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " Nov 04 01:36:06.436729 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.436116 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istio-token\" (UniqueName: \"kubernetes.io/projected/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istio-token\") pod \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " Nov 04 01:36:06.436729 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.436163 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-istio-data\") pod \"eedbad81-5540-45d7-adbb-54afd75aa6e8\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " Nov 04 01:36:06.436729 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.436211 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-workload-certs\") pod \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " Nov 04 01:36:06.436729 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.436247 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-istio-envoy\") pod \"eedbad81-5540-45d7-adbb-54afd75aa6e8\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " Nov 04 01:36:06.436729 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.436278 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-workload-socket\") pod \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\" (UID: \"b42e3b72-1c9f-4a14-88c8-755b3faceb5a\") " Nov 04 01:36:06.436729 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.436333 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-workload-socket\") pod \"eedbad81-5540-45d7-adbb-54afd75aa6e8\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " Nov 04 01:36:06.436729 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.436393 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-credential-socket\") pod \"eedbad81-5540-45d7-adbb-54afd75aa6e8\" (UID: \"eedbad81-5540-45d7-adbb-54afd75aa6e8\") " Nov 04 01:36:06.436729 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.436451 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-workload-certs" pod "eedbad81-5540-45d7-adbb-54afd75aa6e8" (UID: "eedbad81-5540-45d7-adbb-54afd75aa6e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.437389 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.436775 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istiod-ca-cert" pod "b42e3b72-1c9f-4a14-88c8-755b3faceb5a" (UID: "b42e3b72-1c9f-4a14-88c8-755b3faceb5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:36:06.437389 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.436819 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.437389 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.436841 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwlfd\" (UniqueName: \"kubernetes.io/projected/7e59c4bd-a00f-41e1-8d1c-610b9d791374-kube-api-access-fwlfd\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.437389 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.436858 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.437389 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.436873 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.437389 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.436888 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.437389 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.436902 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.437389 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.436915 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.437389 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.436949 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e59c4bd-a00f-41e1-8d1c-610b9d791374-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.437389 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.436963 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e59c4bd-a00f-41e1-8d1c-610b9d791374-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.442132 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.440437 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eedbad81-5540-45d7-adbb-54afd75aa6e8-istio-podinfo" pod "eedbad81-5540-45d7-adbb-54afd75aa6e8" (UID: "eedbad81-5540-45d7-adbb-54afd75aa6e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:36:06.446179 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.445351 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-istio-data" pod "eedbad81-5540-45d7-adbb-54afd75aa6e8" (UID: "eedbad81-5540-45d7-adbb-54afd75aa6e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.446179 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.445665 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eedbad81-5540-45d7-adbb-54afd75aa6e8-istiod-ca-cert" pod "eedbad81-5540-45d7-adbb-54afd75aa6e8" (UID: "eedbad81-5540-45d7-adbb-54afd75aa6e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:36:06.446179 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.445907 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istio-data" pod "b42e3b72-1c9f-4a14-88c8-755b3faceb5a" (UID: "b42e3b72-1c9f-4a14-88c8-755b3faceb5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.448972 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.448692 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eedbad81-5540-45d7-adbb-54afd75aa6e8-istio-token" pod "eedbad81-5540-45d7-adbb-54afd75aa6e8" (UID: "eedbad81-5540-45d7-adbb-54afd75aa6e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:06.452025 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.451163 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istio-token" pod "b42e3b72-1c9f-4a14-88c8-755b3faceb5a" (UID: "b42e3b72-1c9f-4a14-88c8-755b3faceb5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:06.454729 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.454470 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-credential-socket" pod "b42e3b72-1c9f-4a14-88c8-755b3faceb5a" (UID: "b42e3b72-1c9f-4a14-88c8-755b3faceb5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.457582 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.456997 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eedbad81-5540-45d7-adbb-54afd75aa6e8-kube-api-access-ht2ln" pod "eedbad81-5540-45d7-adbb-54afd75aa6e8" (UID: "eedbad81-5540-45d7-adbb-54afd75aa6e8"). InnerVolumeSpecName "kube-api-access-ht2ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:06.458160 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.457981 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-workload-socket" pod "b42e3b72-1c9f-4a14-88c8-755b3faceb5a" (UID: "b42e3b72-1c9f-4a14-88c8-755b3faceb5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.458538 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.458306 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-workload-certs" pod "b42e3b72-1c9f-4a14-88c8-755b3faceb5a" (UID: "b42e3b72-1c9f-4a14-88c8-755b3faceb5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.461671 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.461349 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-istio-envoy" pod "eedbad81-5540-45d7-adbb-54afd75aa6e8" (UID: "eedbad81-5540-45d7-adbb-54afd75aa6e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.470780 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.470729 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-workload-socket" pod "eedbad81-5540-45d7-adbb-54afd75aa6e8" (UID: "eedbad81-5540-45d7-adbb-54afd75aa6e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.471225 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.471137 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8498f949d-mpjs7" event={"ID":"8816836c-a119-4597-9909-778308ea72bc","Type":"ContainerStarted","Data":"522a5c039b2a9d37c7bc3cad0160f9ce50e3bc69ae563cda78dadfc577cbb93b"} Nov 04 01:36:06.479084 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.473993 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-credential-socket" pod "eedbad81-5540-45d7-adbb-54afd75aa6e8" (UID: "eedbad81-5540-45d7-adbb-54afd75aa6e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.534335 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.531558 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istio-envoy" pod "b42e3b72-1c9f-4a14-88c8-755b3faceb5a" (UID: "b42e3b72-1c9f-4a14-88c8-755b3faceb5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.536434 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.536056 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-kube-api-access-vm9qp" pod "b42e3b72-1c9f-4a14-88c8-755b3faceb5a" (UID: "b42e3b72-1c9f-4a14-88c8-755b3faceb5a"). InnerVolumeSpecName "kube-api-access-vm9qp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:06.536958 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.536771 7059 generic.go:356] "Generic (PLEG): container finished" podID="eedbad81-5540-45d7-adbb-54afd75aa6e8" containerID="a3d4097794e797d260b9122d55862b113ff5987deb7606389f7dd8d59ce065d1" exitCode=0 Nov 04 01:36:06.566947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.537071 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" event={"ID":"eedbad81-5540-45d7-adbb-54afd75aa6e8","Type":"ContainerDied","Data":"a3d4097794e797d260b9122d55862b113ff5987deb7606389f7dd8d59ce065d1"} Nov 04 01:36:06.566947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.537137 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" event={"ID":"eedbad81-5540-45d7-adbb-54afd75aa6e8","Type":"ContainerDied","Data":"9336975a2bc1f3c200335a382675672d9f30877ff3cce848e9906e3c7b8a0cef"} Nov 04 01:36:06.566947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.537167 7059 scope.go:122] "RemoveContainer" containerID="a3d4097794e797d260b9122d55862b113ff5987deb7606389f7dd8d59ce065d1" Nov 04 01:36:06.566947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.537319 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9" Nov 04 01:36:06.566947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.556775 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istio-podinfo" pod "b42e3b72-1c9f-4a14-88c8-755b3faceb5a" (UID: "b42e3b72-1c9f-4a14-88c8-755b3faceb5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:36:06.566947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.557512 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.566947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.557537 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.566947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.557554 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm9qp\" (UniqueName: \"kubernetes.io/projected/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-kube-api-access-vm9qp\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.566947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.557570 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht2ln\" (UniqueName: \"kubernetes.io/projected/eedbad81-5540-45d7-adbb-54afd75aa6e8-kube-api-access-ht2ln\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.566947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.557584 7059 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-credential-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.566947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.557599 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.566947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.557612 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.566947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.557627 7059 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eedbad81-5540-45d7-adbb-54afd75aa6e8-istio-podinfo\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.566947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.557641 7059 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eedbad81-5540-45d7-adbb-54afd75aa6e8-istiod-ca-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.566947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.557654 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.566947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.557668 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eedbad81-5540-45d7-adbb-54afd75aa6e8-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.566947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.557682 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.566947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.557696 7059 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-istio-data\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.566947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.557709 7059 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-workload-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.566947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.557723 7059 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-istio-envoy\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.566947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.557738 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42e3b72-1c9f-4a14-88c8-755b3faceb5a-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.566947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.557752 7059 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedbad81-5540-45d7-adbb-54afd75aa6e8-workload-socket\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.628390 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.628360 7059 generic.go:356] "Generic (PLEG): container finished" podID="7e59c4bd-a00f-41e1-8d1c-610b9d791374" containerID="7f68e8df09a677e526e8ba13b3b5bd8a1e9949fa96226e762581834255b5d0ff" exitCode=0 Nov 04 01:36:06.628557 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.628540 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" Nov 04 01:36:06.629575 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.629533 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" event={"ID":"7e59c4bd-a00f-41e1-8d1c-610b9d791374","Type":"ContainerDied","Data":"7f68e8df09a677e526e8ba13b3b5bd8a1e9949fa96226e762581834255b5d0ff"} Nov 04 01:36:06.629677 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.629602 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7" event={"ID":"7e59c4bd-a00f-41e1-8d1c-610b9d791374","Type":"ContainerDied","Data":"5aeccf0750bc51d1b19da0fc77d9464e1e7e30403f0dbd5b841e884f07a68a07"} Nov 04 01:36:06.632270 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.632248 7059 generic.go:356] "Generic (PLEG): container finished" podID="b42e3b72-1c9f-4a14-88c8-755b3faceb5a" containerID="090fb0a7cd2fd554cac2320d82f5c4dda1a6fe7de2c781dcf90517fb71d38819" exitCode=0 Nov 04 01:36:06.632367 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.632335 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" event={"ID":"b42e3b72-1c9f-4a14-88c8-755b3faceb5a","Type":"ContainerDied","Data":"090fb0a7cd2fd554cac2320d82f5c4dda1a6fe7de2c781dcf90517fb71d38819"} Nov 04 01:36:06.632425 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.632378 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" event={"ID":"b42e3b72-1c9f-4a14-88c8-755b3faceb5a","Type":"ContainerDied","Data":"81dcabfb9875cbd94da7aff38f26fe46feee071639d1889da41a6ab209ce2b5d"} Nov 04 01:36:06.632485 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.632448 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499" Nov 04 01:36:06.634190 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.634040 7059 scope.go:122] "RemoveContainer" containerID="a3d4097794e797d260b9122d55862b113ff5987deb7606389f7dd8d59ce065d1" Nov 04 01:36:06.634628 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:36:06.634501 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3d4097794e797d260b9122d55862b113ff5987deb7606389f7dd8d59ce065d1\": container with ID starting with a3d4097794e797d260b9122d55862b113ff5987deb7606389f7dd8d59ce065d1 not found: ID does not exist" containerID="a3d4097794e797d260b9122d55862b113ff5987deb7606389f7dd8d59ce065d1" Nov 04 01:36:06.634628 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.634533 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3d4097794e797d260b9122d55862b113ff5987deb7606389f7dd8d59ce065d1"} err="failed to get container status \"a3d4097794e797d260b9122d55862b113ff5987deb7606389f7dd8d59ce065d1\": rpc error: code = NotFound desc = could not find container \"a3d4097794e797d260b9122d55862b113ff5987deb7606389f7dd8d59ce065d1\": container with ID starting with a3d4097794e797d260b9122d55862b113ff5987deb7606389f7dd8d59ce065d1 not found: ID does not exist" Nov 04 01:36:06.634628 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.634556 7059 scope.go:122] "RemoveContainer" containerID="7f68e8df09a677e526e8ba13b3b5bd8a1e9949fa96226e762581834255b5d0ff" Nov 04 01:36:06.723971 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:36:06.718154 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:36:06.724534 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.724371 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9"] Nov 04 01:36:06.727990 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.727882 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-nd2g9"] Nov 04 01:36:06.732243 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.732219 7059 scope.go:122] "RemoveContainer" containerID="7f68e8df09a677e526e8ba13b3b5bd8a1e9949fa96226e762581834255b5d0ff" Nov 04 01:36:06.732714 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:36:06.732689 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f68e8df09a677e526e8ba13b3b5bd8a1e9949fa96226e762581834255b5d0ff\": container with ID starting with 7f68e8df09a677e526e8ba13b3b5bd8a1e9949fa96226e762581834255b5d0ff not found: ID does not exist" containerID="7f68e8df09a677e526e8ba13b3b5bd8a1e9949fa96226e762581834255b5d0ff" Nov 04 01:36:06.732819 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.732725 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f68e8df09a677e526e8ba13b3b5bd8a1e9949fa96226e762581834255b5d0ff"} err="failed to get container status \"7f68e8df09a677e526e8ba13b3b5bd8a1e9949fa96226e762581834255b5d0ff\": rpc error: code = NotFound desc = could not find container \"7f68e8df09a677e526e8ba13b3b5bd8a1e9949fa96226e762581834255b5d0ff\": container with ID starting with 7f68e8df09a677e526e8ba13b3b5bd8a1e9949fa96226e762581834255b5d0ff not found: ID does not exist" Nov 04 01:36:06.732819 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.732748 7059 scope.go:122] "RemoveContainer" containerID="090fb0a7cd2fd554cac2320d82f5c4dda1a6fe7de2c781dcf90517fb71d38819" Nov 04 01:36:06.822065 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.808646 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499"] Nov 04 01:36:06.822065 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.815572 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-l6499"] Nov 04 01:36:06.822065 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.819955 7059 scope.go:122] "RemoveContainer" containerID="090fb0a7cd2fd554cac2320d82f5c4dda1a6fe7de2c781dcf90517fb71d38819" Nov 04 01:36:06.822065 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:36:06.820418 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"090fb0a7cd2fd554cac2320d82f5c4dda1a6fe7de2c781dcf90517fb71d38819\": container with ID starting with 090fb0a7cd2fd554cac2320d82f5c4dda1a6fe7de2c781dcf90517fb71d38819 not found: ID does not exist" containerID="090fb0a7cd2fd554cac2320d82f5c4dda1a6fe7de2c781dcf90517fb71d38819" Nov 04 01:36:06.822065 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.820474 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"090fb0a7cd2fd554cac2320d82f5c4dda1a6fe7de2c781dcf90517fb71d38819"} err="failed to get container status \"090fb0a7cd2fd554cac2320d82f5c4dda1a6fe7de2c781dcf90517fb71d38819\": rpc error: code = NotFound desc = could not find container \"090fb0a7cd2fd554cac2320d82f5c4dda1a6fe7de2c781dcf90517fb71d38819\": container with ID starting with 090fb0a7cd2fd554cac2320d82f5c4dda1a6fe7de2c781dcf90517fb71d38819 not found: ID does not exist" Nov 04 01:36:06.831454 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.831422 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7"] Nov 04 01:36:06.834019 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:06.833998 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8pjd7"] Nov 04 01:36:07.211949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:07.205566 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e59c4bd-a00f-41e1-8d1c-610b9d791374" path="/var/lib/kubelet/pods/7e59c4bd-a00f-41e1-8d1c-610b9d791374/volumes" Nov 04 01:36:07.213464 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:07.212437 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b42e3b72-1c9f-4a14-88c8-755b3faceb5a" path="/var/lib/kubelet/pods/b42e3b72-1c9f-4a14-88c8-755b3faceb5a/volumes" Nov 04 01:36:07.213464 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:07.213064 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eedbad81-5540-45d7-adbb-54afd75aa6e8" path="/var/lib/kubelet/pods/eedbad81-5540-45d7-adbb-54afd75aa6e8/volumes" Nov 04 01:36:07.658961 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:07.653876 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8498f949d-mpjs7" event={"ID":"8816836c-a119-4597-9909-778308ea72bc","Type":"ContainerStarted","Data":"4fba21fbad290beafa15632c651a4d2b66e85e1efa57d90603d7858bf0091b95"} Nov 04 01:36:07.658961 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:07.654260 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8498f949d-mpjs7" Nov 04 01:36:07.674572 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:07.674507 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-8498f949d-mpjs7" podStartSLOduration=1.562612959 podStartE2EDuration="2.674489356s" totalImagesPullingTime="1.111876397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:36:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:36:07.670670093 +0000 UTC m=+2252.612518996" watchObservedRunningTime="2026-11-04 01:36:07.674489356 +0000 UTC m=+2252.616338261" Nov 04 01:36:18.665053 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:18.664978 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8498f949d-mpjs7" Nov 04 01:36:25.487099 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:25.487066 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4cprq"] Nov 04 01:36:25.487643 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:25.487610 7059 cpu_manager.go:414] "containerMap: removing container" podUID="7e59c4bd-a00f-41e1-8d1c-610b9d791374" containerName="istio-proxy" Nov 04 01:36:25.487643 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:25.487627 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e59c4bd-a00f-41e1-8d1c-610b9d791374" containerName="istio-proxy" Nov 04 01:36:25.487903 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:25.487660 7059 cpu_manager.go:414] "containerMap: removing container" podUID="b42e3b72-1c9f-4a14-88c8-755b3faceb5a" containerName="istio-proxy" Nov 04 01:36:25.487903 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:25.487668 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b42e3b72-1c9f-4a14-88c8-755b3faceb5a" containerName="istio-proxy" Nov 04 01:36:25.487903 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:25.487677 7059 cpu_manager.go:414] "containerMap: removing container" podUID="eedbad81-5540-45d7-adbb-54afd75aa6e8" containerName="istio-proxy" Nov 04 01:36:25.487903 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:25.487683 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eedbad81-5540-45d7-adbb-54afd75aa6e8" containerName="istio-proxy" Nov 04 01:36:25.487903 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:25.487796 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e59c4bd-a00f-41e1-8d1c-610b9d791374" containerName="istio-proxy" Nov 04 01:36:25.487903 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:25.487812 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="b42e3b72-1c9f-4a14-88c8-755b3faceb5a" containerName="istio-proxy" Nov 04 01:36:25.487903 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:25.487822 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="eedbad81-5540-45d7-adbb-54afd75aa6e8" containerName="istio-proxy" Nov 04 01:36:25.491679 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:25.491661 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4cprq" Nov 04 01:36:25.500881 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:25.500852 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:36:25.500993 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:25.500910 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:36:25.502409 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:25.502368 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4cprq"] Nov 04 01:36:25.597279 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:25.597245 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c9538e-6964-4310-839a-9d8c2774501f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-4cprq\" (UID: \"88c9538e-6964-4310-839a-9d8c2774501f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4cprq" Nov 04 01:36:25.597454 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:25.597347 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c9538e-6964-4310-839a-9d8c2774501f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-4cprq\" (UID: \"88c9538e-6964-4310-839a-9d8c2774501f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4cprq" Nov 04 01:36:25.597500 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:25.597467 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6t99\" (UniqueName: \"kubernetes.io/projected/88c9538e-6964-4310-839a-9d8c2774501f-kube-api-access-x6t99\") pod \"maas-api-revoke-sub-85237e45-4cprq\" (UID: \"88c9538e-6964-4310-839a-9d8c2774501f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4cprq" Nov 04 01:36:25.698534 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:25.698497 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c9538e-6964-4310-839a-9d8c2774501f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-4cprq\" (UID: \"88c9538e-6964-4310-839a-9d8c2774501f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4cprq" Nov 04 01:36:25.698700 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:25.698603 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6t99\" (UniqueName: \"kubernetes.io/projected/88c9538e-6964-4310-839a-9d8c2774501f-kube-api-access-x6t99\") pod \"maas-api-revoke-sub-85237e45-4cprq\" (UID: \"88c9538e-6964-4310-839a-9d8c2774501f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4cprq" Nov 04 01:36:25.698700 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:25.698677 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c9538e-6964-4310-839a-9d8c2774501f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-4cprq\" (UID: \"88c9538e-6964-4310-839a-9d8c2774501f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4cprq" Nov 04 01:36:25.699180 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:25.699158 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c9538e-6964-4310-839a-9d8c2774501f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-4cprq\" (UID: \"88c9538e-6964-4310-839a-9d8c2774501f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4cprq" Nov 04 01:36:25.701211 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:25.701190 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c9538e-6964-4310-839a-9d8c2774501f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-4cprq\" (UID: \"88c9538e-6964-4310-839a-9d8c2774501f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4cprq" Nov 04 01:36:25.707652 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:25.707615 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6t99\" (UniqueName: \"kubernetes.io/projected/88c9538e-6964-4310-839a-9d8c2774501f-kube-api-access-x6t99\") pod \"maas-api-revoke-sub-85237e45-4cprq\" (UID: \"88c9538e-6964-4310-839a-9d8c2774501f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4cprq" Nov 04 01:36:25.808525 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:25.808451 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4cprq" Nov 04 01:36:26.088765 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:26.088736 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4cprq"] Nov 04 01:36:26.090573 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:36:26.090546 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88c9538e_6964_4310_839a_9d8c2774501f.slice/crio-55d55f62130e30f560968408a787d526aa7698cf6a374a807248a0ca75e2de7a WatchSource:0}: Error finding container 55d55f62130e30f560968408a787d526aa7698cf6a374a807248a0ca75e2de7a: Status 404 returned error can't find the container with id 55d55f62130e30f560968408a787d526aa7698cf6a374a807248a0ca75e2de7a Nov 04 01:36:26.771488 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:26.771455 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4cprq" event={"ID":"88c9538e-6964-4310-839a-9d8c2774501f","Type":"ContainerStarted","Data":"55d55f62130e30f560968408a787d526aa7698cf6a374a807248a0ca75e2de7a"} Nov 04 01:36:30.798189 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:30.798106 7059 generic.go:356] "Generic (PLEG): container finished" podID="88c9538e-6964-4310-839a-9d8c2774501f" containerID="f8f75d5982f067a50ce6c6dd1725127d7f1accd3e6d4505dc9fac06ff35423b1" exitCode=0 Nov 04 01:36:30.798551 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:30.798177 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4cprq" event={"ID":"88c9538e-6964-4310-839a-9d8c2774501f","Type":"ContainerDied","Data":"f8f75d5982f067a50ce6c6dd1725127d7f1accd3e6d4505dc9fac06ff35423b1"} Nov 04 01:36:31.951084 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:31.951059 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4cprq" Nov 04 01:36:32.064785 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:32.064692 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88c9538e-6964-4310-839a-9d8c2774501f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c9538e-6964-4310-839a-9d8c2774501f-maas-api-openshift-ca\") pod \"88c9538e-6964-4310-839a-9d8c2774501f\" (UID: \"88c9538e-6964-4310-839a-9d8c2774501f\") " Nov 04 01:36:32.064785 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:32.064759 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88c9538e-6964-4310-839a-9d8c2774501f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c9538e-6964-4310-839a-9d8c2774501f-maas-api-xks-ca\") pod \"88c9538e-6964-4310-839a-9d8c2774501f\" (UID: \"88c9538e-6964-4310-839a-9d8c2774501f\") " Nov 04 01:36:32.065060 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:32.064830 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88c9538e-6964-4310-839a-9d8c2774501f-kube-api-access-x6t99\" (UniqueName: \"kubernetes.io/projected/88c9538e-6964-4310-839a-9d8c2774501f-kube-api-access-x6t99\") pod \"88c9538e-6964-4310-839a-9d8c2774501f\" (UID: \"88c9538e-6964-4310-839a-9d8c2774501f\") " Nov 04 01:36:32.065156 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:32.065126 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88c9538e-6964-4310-839a-9d8c2774501f-maas-api-openshift-ca" pod "88c9538e-6964-4310-839a-9d8c2774501f" (UID: "88c9538e-6964-4310-839a-9d8c2774501f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:36:32.065364 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:32.065349 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c9538e-6964-4310-839a-9d8c2774501f-maas-api-openshift-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:32.067091 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:32.067061 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88c9538e-6964-4310-839a-9d8c2774501f-maas-api-xks-ca" pod "88c9538e-6964-4310-839a-9d8c2774501f" (UID: "88c9538e-6964-4310-839a-9d8c2774501f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:36:32.067198 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:32.067120 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c9538e-6964-4310-839a-9d8c2774501f-kube-api-access-x6t99" pod "88c9538e-6964-4310-839a-9d8c2774501f" (UID: "88c9538e-6964-4310-839a-9d8c2774501f"). InnerVolumeSpecName "kube-api-access-x6t99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:32.166241 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:32.166215 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c9538e-6964-4310-839a-9d8c2774501f-maas-api-xks-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:32.166241 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:32.166238 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6t99\" (UniqueName: \"kubernetes.io/projected/88c9538e-6964-4310-839a-9d8c2774501f-kube-api-access-x6t99\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:32.812483 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:32.812450 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4cprq" Nov 04 01:36:32.812653 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:32.812501 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4cprq" event={"ID":"88c9538e-6964-4310-839a-9d8c2774501f","Type":"ContainerDied","Data":"55d55f62130e30f560968408a787d526aa7698cf6a374a807248a0ca75e2de7a"} Nov 04 01:36:32.812653 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:32.812586 7059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55d55f62130e30f560968408a787d526aa7698cf6a374a807248a0ca75e2de7a" Nov 04 01:36:36.278599 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:36.278571 7059 scope.go:122] "RemoveContainer" containerID="619ea51e3ee310ce0c365fead6098790c703811ccd4ad03253cde74cda180a5a" Nov 04 01:36:36.287989 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:36.287971 7059 scope.go:122] "RemoveContainer" containerID="b647b5c6a8b6e88bdceaa9d566717d564fc92413ac008cab7dc540c040280205" Nov 04 01:36:36.478916 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:36:36.478885 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:36:37.203407 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:37.203372 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8498f949d-mpjs7"] Nov 04 01:36:37.203601 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:37.203558 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-8498f949d-mpjs7" podUID="8816836c-a119-4597-9909-778308ea72bc" containerName="manager" containerID="cri-o://4fba21fbad290beafa15632c651a4d2b66e85e1efa57d90603d7858bf0091b95" gracePeriod=30 Nov 04 01:36:37.466021 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:37.465967 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8498f949d-mpjs7" Nov 04 01:36:37.623085 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:37.623050 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8816836c-a119-4597-9909-778308ea72bc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8816836c-a119-4597-9909-778308ea72bc-metrics-certs\") pod \"8816836c-a119-4597-9909-778308ea72bc\" (UID: \"8816836c-a119-4597-9909-778308ea72bc\") " Nov 04 01:36:37.623232 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:37.623117 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8816836c-a119-4597-9909-778308ea72bc-webhook-cert\" (UniqueName: \"kubernetes.io/secret/8816836c-a119-4597-9909-778308ea72bc-webhook-cert\") pod \"8816836c-a119-4597-9909-778308ea72bc\" (UID: \"8816836c-a119-4597-9909-778308ea72bc\") " Nov 04 01:36:37.623232 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:37.623143 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8816836c-a119-4597-9909-778308ea72bc-kube-api-access-46hwj\" (UniqueName: \"kubernetes.io/projected/8816836c-a119-4597-9909-778308ea72bc-kube-api-access-46hwj\") pod \"8816836c-a119-4597-9909-778308ea72bc\" (UID: \"8816836c-a119-4597-9909-778308ea72bc\") " Nov 04 01:36:37.625180 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:37.625150 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8816836c-a119-4597-9909-778308ea72bc-metrics-certs" pod "8816836c-a119-4597-9909-778308ea72bc" (UID: "8816836c-a119-4597-9909-778308ea72bc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:36:37.625307 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:37.625270 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8816836c-a119-4597-9909-778308ea72bc-kube-api-access-46hwj" pod "8816836c-a119-4597-9909-778308ea72bc" (UID: "8816836c-a119-4597-9909-778308ea72bc"). InnerVolumeSpecName "kube-api-access-46hwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:37.625307 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:37.625278 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8816836c-a119-4597-9909-778308ea72bc-webhook-cert" pod "8816836c-a119-4597-9909-778308ea72bc" (UID: "8816836c-a119-4597-9909-778308ea72bc"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:36:37.724125 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:37.724064 7059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8816836c-a119-4597-9909-778308ea72bc-metrics-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:37.724125 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:37.724086 7059 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8816836c-a119-4597-9909-778308ea72bc-webhook-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:37.724125 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:37.724097 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46hwj\" (UniqueName: \"kubernetes.io/projected/8816836c-a119-4597-9909-778308ea72bc-kube-api-access-46hwj\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:37.841440 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:37.841408 7059 generic.go:356] "Generic (PLEG): container finished" podID="8816836c-a119-4597-9909-778308ea72bc" containerID="4fba21fbad290beafa15632c651a4d2b66e85e1efa57d90603d7858bf0091b95" exitCode=0 Nov 04 01:36:37.841440 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:37.841443 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8498f949d-mpjs7" event={"ID":"8816836c-a119-4597-9909-778308ea72bc","Type":"ContainerDied","Data":"4fba21fbad290beafa15632c651a4d2b66e85e1efa57d90603d7858bf0091b95"} Nov 04 01:36:37.841628 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:37.841454 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8498f949d-mpjs7" Nov 04 01:36:37.841628 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:37.841506 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8498f949d-mpjs7" event={"ID":"8816836c-a119-4597-9909-778308ea72bc","Type":"ContainerDied","Data":"522a5c039b2a9d37c7bc3cad0160f9ce50e3bc69ae563cda78dadfc577cbb93b"} Nov 04 01:36:37.841628 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:37.841530 7059 scope.go:122] "RemoveContainer" containerID="4fba21fbad290beafa15632c651a4d2b66e85e1efa57d90603d7858bf0091b95" Nov 04 01:36:37.852466 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:37.852446 7059 scope.go:122] "RemoveContainer" containerID="4fba21fbad290beafa15632c651a4d2b66e85e1efa57d90603d7858bf0091b95" Nov 04 01:36:37.852778 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:36:37.852760 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fba21fbad290beafa15632c651a4d2b66e85e1efa57d90603d7858bf0091b95\": container with ID starting with 4fba21fbad290beafa15632c651a4d2b66e85e1efa57d90603d7858bf0091b95 not found: ID does not exist" containerID="4fba21fbad290beafa15632c651a4d2b66e85e1efa57d90603d7858bf0091b95" Nov 04 01:36:37.852834 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:37.852787 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fba21fbad290beafa15632c651a4d2b66e85e1efa57d90603d7858bf0091b95"} err="failed to get container status \"4fba21fbad290beafa15632c651a4d2b66e85e1efa57d90603d7858bf0091b95\": rpc error: code = NotFound desc = could not find container \"4fba21fbad290beafa15632c651a4d2b66e85e1efa57d90603d7858bf0091b95\": container with ID starting with 4fba21fbad290beafa15632c651a4d2b66e85e1efa57d90603d7858bf0091b95 not found: ID does not exist" Nov 04 01:36:37.863646 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:37.863621 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8498f949d-mpjs7"] Nov 04 01:36:37.868900 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:37.868880 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8498f949d-mpjs7"] Nov 04 01:36:39.076220 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:39.076186 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8498f949d-j6mmc"] Nov 04 01:36:39.076741 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:39.076618 7059 cpu_manager.go:414] "containerMap: removing container" podUID="88c9538e-6964-4310-839a-9d8c2774501f" containerName="revoke-keys" Nov 04 01:36:39.076741 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:39.076635 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88c9538e-6964-4310-839a-9d8c2774501f" containerName="revoke-keys" Nov 04 01:36:39.076741 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:39.076666 7059 cpu_manager.go:414] "containerMap: removing container" podUID="8816836c-a119-4597-9909-778308ea72bc" containerName="manager" Nov 04 01:36:39.076741 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:39.076674 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8816836c-a119-4597-9909-778308ea72bc" containerName="manager" Nov 04 01:36:39.077075 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:39.076915 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8816836c-a119-4597-9909-778308ea72bc" containerName="manager" Nov 04 01:36:39.077075 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:39.076997 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="88c9538e-6964-4310-839a-9d8c2774501f" containerName="revoke-keys" Nov 04 01:36:39.081863 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:39.081846 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8498f949d-j6mmc" Nov 04 01:36:39.086735 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:39.086660 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 04 01:36:39.088014 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:39.087991 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 04 01:36:39.088112 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:39.088022 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 04 01:36:39.088251 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:39.088234 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nlglx\"" Nov 04 01:36:39.091332 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:39.091311 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8498f949d-j6mmc"] Nov 04 01:36:39.176161 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:39.176134 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8816836c-a119-4597-9909-778308ea72bc" path="/var/lib/kubelet/pods/8816836c-a119-4597-9909-778308ea72bc/volumes" Nov 04 01:36:39.239847 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:39.239816 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl5h4\" (UniqueName: \"kubernetes.io/projected/fc16e4aa-a8ba-42f2-8918-8e8c3ebcb78b-kube-api-access-tl5h4\") pod \"maas-controller-8498f949d-j6mmc\" (UID: \"fc16e4aa-a8ba-42f2-8918-8e8c3ebcb78b\") " pod="opendatahub/maas-controller-8498f949d-j6mmc" Nov 04 01:36:39.239994 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:39.239866 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc16e4aa-a8ba-42f2-8918-8e8c3ebcb78b-metrics-certs\") pod \"maas-controller-8498f949d-j6mmc\" (UID: \"fc16e4aa-a8ba-42f2-8918-8e8c3ebcb78b\") " pod="opendatahub/maas-controller-8498f949d-j6mmc" Nov 04 01:36:39.239994 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:39.239965 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc16e4aa-a8ba-42f2-8918-8e8c3ebcb78b-webhook-cert\") pod \"maas-controller-8498f949d-j6mmc\" (UID: \"fc16e4aa-a8ba-42f2-8918-8e8c3ebcb78b\") " pod="opendatahub/maas-controller-8498f949d-j6mmc" Nov 04 01:36:39.341306 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:39.341239 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc16e4aa-a8ba-42f2-8918-8e8c3ebcb78b-webhook-cert\") pod \"maas-controller-8498f949d-j6mmc\" (UID: \"fc16e4aa-a8ba-42f2-8918-8e8c3ebcb78b\") " pod="opendatahub/maas-controller-8498f949d-j6mmc" Nov 04 01:36:39.341414 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:39.341352 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl5h4\" (UniqueName: \"kubernetes.io/projected/fc16e4aa-a8ba-42f2-8918-8e8c3ebcb78b-kube-api-access-tl5h4\") pod \"maas-controller-8498f949d-j6mmc\" (UID: \"fc16e4aa-a8ba-42f2-8918-8e8c3ebcb78b\") " pod="opendatahub/maas-controller-8498f949d-j6mmc" Nov 04 01:36:39.341414 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:39.341389 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc16e4aa-a8ba-42f2-8918-8e8c3ebcb78b-metrics-certs\") pod \"maas-controller-8498f949d-j6mmc\" (UID: \"fc16e4aa-a8ba-42f2-8918-8e8c3ebcb78b\") " pod="opendatahub/maas-controller-8498f949d-j6mmc" Nov 04 01:36:39.343789 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:39.343767 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc16e4aa-a8ba-42f2-8918-8e8c3ebcb78b-metrics-certs\") pod \"maas-controller-8498f949d-j6mmc\" (UID: \"fc16e4aa-a8ba-42f2-8918-8e8c3ebcb78b\") " pod="opendatahub/maas-controller-8498f949d-j6mmc" Nov 04 01:36:39.343950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:39.343906 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc16e4aa-a8ba-42f2-8918-8e8c3ebcb78b-webhook-cert\") pod \"maas-controller-8498f949d-j6mmc\" (UID: \"fc16e4aa-a8ba-42f2-8918-8e8c3ebcb78b\") " pod="opendatahub/maas-controller-8498f949d-j6mmc" Nov 04 01:36:39.351426 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:39.351403 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl5h4\" (UniqueName: \"kubernetes.io/projected/fc16e4aa-a8ba-42f2-8918-8e8c3ebcb78b-kube-api-access-tl5h4\") pod \"maas-controller-8498f949d-j6mmc\" (UID: \"fc16e4aa-a8ba-42f2-8918-8e8c3ebcb78b\") " pod="opendatahub/maas-controller-8498f949d-j6mmc" Nov 04 01:36:39.396409 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:39.396394 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8498f949d-j6mmc" Nov 04 01:36:39.678250 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:39.678226 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8498f949d-j6mmc"] Nov 04 01:36:39.680243 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:36:39.680211 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc16e4aa_a8ba_42f2_8918_8e8c3ebcb78b.slice/crio-45bc437ef2d74be4376537de81e1c18816f1a9a225ec30f12cf5084726518048 WatchSource:0}: Error finding container 45bc437ef2d74be4376537de81e1c18816f1a9a225ec30f12cf5084726518048: Status 404 returned error can't find the container with id 45bc437ef2d74be4376537de81e1c18816f1a9a225ec30f12cf5084726518048 Nov 04 01:36:39.856500 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:39.856459 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8498f949d-j6mmc" event={"ID":"fc16e4aa-a8ba-42f2-8918-8e8c3ebcb78b","Type":"ContainerStarted","Data":"45bc437ef2d74be4376537de81e1c18816f1a9a225ec30f12cf5084726518048"} Nov 04 01:36:40.864644 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:40.864613 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8498f949d-j6mmc" event={"ID":"fc16e4aa-a8ba-42f2-8918-8e8c3ebcb78b","Type":"ContainerStarted","Data":"0245c19ba17c1526d8a67288a9753dfd8bf12b678938892c98d42262c86b65ee"} Nov 04 01:36:40.865078 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:40.864794 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8498f949d-j6mmc" Nov 04 01:36:40.883840 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:40.883790 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-8498f949d-j6mmc" podStartSLOduration=1.355948911 podStartE2EDuration="1.883778282s" totalImagesPullingTime="527.829371ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:36:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:36:40.882536609 +0000 UTC m=+2285.824385514" watchObservedRunningTime="2026-11-04 01:36:40.883778282 +0000 UTC m=+2285.825627185" Nov 04 01:36:51.871980 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:36:51.871948 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8498f949d-j6mmc" Nov 04 01:37:06.480470 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:37:06.480415 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:37:19.109453 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:19.109419 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-b94h2"] Nov 04 01:37:19.116660 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:19.116640 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-b94h2" Nov 04 01:37:19.120593 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:19.120568 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:37:19.121379 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:19.121357 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:37:19.121729 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:19.121705 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-b94h2"] Nov 04 01:37:19.244888 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:19.244856 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bba91f06-160f-4e53-a1f4-16c4b390db33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-b94h2\" (UID: \"bba91f06-160f-4e53-a1f4-16c4b390db33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-b94h2" Nov 04 01:37:19.245063 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:19.244892 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2fxh\" (UniqueName: \"kubernetes.io/projected/bba91f06-160f-4e53-a1f4-16c4b390db33-kube-api-access-m2fxh\") pod \"maas-api-revoke-sub-c1e271a9-b94h2\" (UID: \"bba91f06-160f-4e53-a1f4-16c4b390db33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-b94h2" Nov 04 01:37:19.245107 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:19.245069 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bba91f06-160f-4e53-a1f4-16c4b390db33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-b94h2\" (UID: \"bba91f06-160f-4e53-a1f4-16c4b390db33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-b94h2" Nov 04 01:37:19.346295 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:19.346265 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bba91f06-160f-4e53-a1f4-16c4b390db33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-b94h2\" (UID: \"bba91f06-160f-4e53-a1f4-16c4b390db33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-b94h2" Nov 04 01:37:19.346464 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:19.346298 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2fxh\" (UniqueName: \"kubernetes.io/projected/bba91f06-160f-4e53-a1f4-16c4b390db33-kube-api-access-m2fxh\") pod \"maas-api-revoke-sub-c1e271a9-b94h2\" (UID: \"bba91f06-160f-4e53-a1f4-16c4b390db33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-b94h2" Nov 04 01:37:19.346464 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:19.346368 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bba91f06-160f-4e53-a1f4-16c4b390db33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-b94h2\" (UID: \"bba91f06-160f-4e53-a1f4-16c4b390db33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-b94h2" Nov 04 01:37:19.347103 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:19.347074 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bba91f06-160f-4e53-a1f4-16c4b390db33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-b94h2\" (UID: \"bba91f06-160f-4e53-a1f4-16c4b390db33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-b94h2" Nov 04 01:37:19.348891 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:19.348867 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bba91f06-160f-4e53-a1f4-16c4b390db33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-b94h2\" (UID: \"bba91f06-160f-4e53-a1f4-16c4b390db33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-b94h2" Nov 04 01:37:19.356559 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:19.356538 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2fxh\" (UniqueName: \"kubernetes.io/projected/bba91f06-160f-4e53-a1f4-16c4b390db33-kube-api-access-m2fxh\") pod \"maas-api-revoke-sub-c1e271a9-b94h2\" (UID: \"bba91f06-160f-4e53-a1f4-16c4b390db33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-b94h2" Nov 04 01:37:19.433414 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:19.433395 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-b94h2" Nov 04 01:37:19.706733 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:19.706709 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-b94h2"] Nov 04 01:37:19.708532 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:37:19.708502 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba91f06_160f_4e53_a1f4_16c4b390db33.slice/crio-67c484d488685d67738f3196200248f7d4a60259cbdea01adee8d4d855cf39c8 WatchSource:0}: Error finding container 67c484d488685d67738f3196200248f7d4a60259cbdea01adee8d4d855cf39c8: Status 404 returned error can't find the container with id 67c484d488685d67738f3196200248f7d4a60259cbdea01adee8d4d855cf39c8 Nov 04 01:37:20.107523 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:20.107448 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-b94h2" event={"ID":"bba91f06-160f-4e53-a1f4-16c4b390db33","Type":"ContainerStarted","Data":"dd63e3327c51bbab4f370f4665c31c39bd216173958ce33fd25f8ddf4c25195b"} Nov 04 01:37:20.107523 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:20.107507 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-b94h2" event={"ID":"bba91f06-160f-4e53-a1f4-16c4b390db33","Type":"ContainerStarted","Data":"67c484d488685d67738f3196200248f7d4a60259cbdea01adee8d4d855cf39c8"} Nov 04 01:37:20.125895 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:20.125854 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-b94h2" podStartSLOduration=1.125839875 podStartE2EDuration="1.125839875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:37:20.123767347 +0000 UTC m=+2325.065616250" watchObservedRunningTime="2026-11-04 01:37:20.125839875 +0000 UTC m=+2325.067688863" Nov 04 01:37:21.114240 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:21.114210 7059 generic.go:356] "Generic (PLEG): container finished" podID="bba91f06-160f-4e53-a1f4-16c4b390db33" containerID="dd63e3327c51bbab4f370f4665c31c39bd216173958ce33fd25f8ddf4c25195b" exitCode=0 Nov 04 01:37:21.114414 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:21.114300 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-b94h2" event={"ID":"bba91f06-160f-4e53-a1f4-16c4b390db33","Type":"ContainerDied","Data":"dd63e3327c51bbab4f370f4665c31c39bd216173958ce33fd25f8ddf4c25195b"} Nov 04 01:37:22.253555 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:22.253531 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-b94h2" Nov 04 01:37:22.374422 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:22.374361 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bba91f06-160f-4e53-a1f4-16c4b390db33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bba91f06-160f-4e53-a1f4-16c4b390db33-maas-api-xks-ca\") pod \"bba91f06-160f-4e53-a1f4-16c4b390db33\" (UID: \"bba91f06-160f-4e53-a1f4-16c4b390db33\") " Nov 04 01:37:22.374422 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:22.374397 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bba91f06-160f-4e53-a1f4-16c4b390db33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bba91f06-160f-4e53-a1f4-16c4b390db33-maas-api-openshift-ca\") pod \"bba91f06-160f-4e53-a1f4-16c4b390db33\" (UID: \"bba91f06-160f-4e53-a1f4-16c4b390db33\") " Nov 04 01:37:22.374630 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:22.374434 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bba91f06-160f-4e53-a1f4-16c4b390db33-kube-api-access-m2fxh\" (UniqueName: \"kubernetes.io/projected/bba91f06-160f-4e53-a1f4-16c4b390db33-kube-api-access-m2fxh\") pod \"bba91f06-160f-4e53-a1f4-16c4b390db33\" (UID: \"bba91f06-160f-4e53-a1f4-16c4b390db33\") " Nov 04 01:37:22.374724 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:22.374699 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bba91f06-160f-4e53-a1f4-16c4b390db33-maas-api-openshift-ca" pod "bba91f06-160f-4e53-a1f4-16c4b390db33" (UID: "bba91f06-160f-4e53-a1f4-16c4b390db33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:37:22.376597 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:22.376559 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bba91f06-160f-4e53-a1f4-16c4b390db33-maas-api-xks-ca" pod "bba91f06-160f-4e53-a1f4-16c4b390db33" (UID: "bba91f06-160f-4e53-a1f4-16c4b390db33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:37:22.376597 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:22.376576 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba91f06-160f-4e53-a1f4-16c4b390db33-kube-api-access-m2fxh" pod "bba91f06-160f-4e53-a1f4-16c4b390db33" (UID: "bba91f06-160f-4e53-a1f4-16c4b390db33"). InnerVolumeSpecName "kube-api-access-m2fxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:37:22.475220 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:22.475195 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bba91f06-160f-4e53-a1f4-16c4b390db33-maas-api-xks-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:22.475220 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:22.475219 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bba91f06-160f-4e53-a1f4-16c4b390db33-maas-api-openshift-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:22.475349 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:22.475230 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2fxh\" (UniqueName: \"kubernetes.io/projected/bba91f06-160f-4e53-a1f4-16c4b390db33-kube-api-access-m2fxh\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:23.126201 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:23.126171 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-b94h2" Nov 04 01:37:23.126366 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:23.126202 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-b94h2" event={"ID":"bba91f06-160f-4e53-a1f4-16c4b390db33","Type":"ContainerDied","Data":"67c484d488685d67738f3196200248f7d4a60259cbdea01adee8d4d855cf39c8"} Nov 04 01:37:23.126366 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:23.126269 7059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67c484d488685d67738f3196200248f7d4a60259cbdea01adee8d4d855cf39c8" Nov 04 01:37:34.232075 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:34.232040 7059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-ddd7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:37:34.232496 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:34.232100 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ddd7x" podUID="26881d61-360e-43bb-83f8-3122e0026e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:37:34.236032 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:34.236009 7059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bprgn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:37:34.236089 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:34.236058 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bprgn" podUID="5e65c0d9-95d9-43a6-b77f-bd41b3fd6383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:37:36.186537 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:36.185769 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m2z56"] Nov 04 01:37:36.186537 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:36.186450 7059 cpu_manager.go:414] "containerMap: removing container" podUID="bba91f06-160f-4e53-a1f4-16c4b390db33" containerName="revoke-keys" Nov 04 01:37:36.186537 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:36.186471 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bba91f06-160f-4e53-a1f4-16c4b390db33" containerName="revoke-keys" Nov 04 01:37:36.187151 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:36.186624 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="bba91f06-160f-4e53-a1f4-16c4b390db33" containerName="revoke-keys" Nov 04 01:37:36.190134 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:36.190112 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m2z56" Nov 04 01:37:36.196838 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:36.196814 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:37:36.197073 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:36.197055 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:37:36.197997 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:36.197972 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m2z56"] Nov 04 01:37:36.234304 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:36.234259 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmbf2\" (UniqueName: \"kubernetes.io/projected/2a40f207-413e-48d0-b971-f350aae4bbd6-kube-api-access-kmbf2\") pod \"maas-api-revoke-sub-2e1f5b5f-m2z56\" (UID: \"2a40f207-413e-48d0-b971-f350aae4bbd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m2z56" Nov 04 01:37:36.234498 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:36.234480 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a40f207-413e-48d0-b971-f350aae4bbd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-m2z56\" (UID: \"2a40f207-413e-48d0-b971-f350aae4bbd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m2z56" Nov 04 01:37:36.234662 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:36.234632 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a40f207-413e-48d0-b971-f350aae4bbd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-m2z56\" (UID: \"2a40f207-413e-48d0-b971-f350aae4bbd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m2z56" Nov 04 01:37:36.335995 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:36.335961 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a40f207-413e-48d0-b971-f350aae4bbd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-m2z56\" (UID: \"2a40f207-413e-48d0-b971-f350aae4bbd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m2z56" Nov 04 01:37:36.336186 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:36.336044 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a40f207-413e-48d0-b971-f350aae4bbd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-m2z56\" (UID: \"2a40f207-413e-48d0-b971-f350aae4bbd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m2z56" Nov 04 01:37:36.336186 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:36.336078 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmbf2\" (UniqueName: \"kubernetes.io/projected/2a40f207-413e-48d0-b971-f350aae4bbd6-kube-api-access-kmbf2\") pod \"maas-api-revoke-sub-2e1f5b5f-m2z56\" (UID: \"2a40f207-413e-48d0-b971-f350aae4bbd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m2z56" Nov 04 01:37:36.336706 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:36.336684 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a40f207-413e-48d0-b971-f350aae4bbd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-m2z56\" (UID: \"2a40f207-413e-48d0-b971-f350aae4bbd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m2z56" Nov 04 01:37:36.338528 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:36.338509 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a40f207-413e-48d0-b971-f350aae4bbd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-m2z56\" (UID: \"2a40f207-413e-48d0-b971-f350aae4bbd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m2z56" Nov 04 01:37:36.344427 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:36.344405 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmbf2\" (UniqueName: \"kubernetes.io/projected/2a40f207-413e-48d0-b971-f350aae4bbd6-kube-api-access-kmbf2\") pod \"maas-api-revoke-sub-2e1f5b5f-m2z56\" (UID: \"2a40f207-413e-48d0-b971-f350aae4bbd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m2z56" Nov 04 01:37:36.480412 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:37:36.480332 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:37:36.508155 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:36.508131 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m2z56" Nov 04 01:37:36.782639 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:36.782610 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m2z56"] Nov 04 01:37:36.783971 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:37:36.783945 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a40f207_413e_48d0_b971_f350aae4bbd6.slice/crio-cfa4dce6bf5be994c3b81226a942cf3fcf14e82c92a10e7320a9c70b2a124306 WatchSource:0}: Error finding container cfa4dce6bf5be994c3b81226a942cf3fcf14e82c92a10e7320a9c70b2a124306: Status 404 returned error can't find the container with id cfa4dce6bf5be994c3b81226a942cf3fcf14e82c92a10e7320a9c70b2a124306 Nov 04 01:37:37.246222 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:37.246174 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m2z56" event={"ID":"2a40f207-413e-48d0-b971-f350aae4bbd6","Type":"ContainerStarted","Data":"5a63f075d63647f98274aa808935adb4b020ab8606837b99d9af9bfd75272d2b"} Nov 04 01:37:37.246222 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:37.246225 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m2z56" event={"ID":"2a40f207-413e-48d0-b971-f350aae4bbd6","Type":"ContainerStarted","Data":"cfa4dce6bf5be994c3b81226a942cf3fcf14e82c92a10e7320a9c70b2a124306"} Nov 04 01:37:37.266144 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:37.266096 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m2z56" podStartSLOduration=1.26608373 podStartE2EDuration="1.26608373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:37:37.264181063 +0000 UTC m=+2342.206029969" watchObservedRunningTime="2026-11-04 01:37:37.26608373 +0000 UTC m=+2342.207932633" Nov 04 01:37:38.253001 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:38.252965 7059 generic.go:356] "Generic (PLEG): container finished" podID="2a40f207-413e-48d0-b971-f350aae4bbd6" containerID="5a63f075d63647f98274aa808935adb4b020ab8606837b99d9af9bfd75272d2b" exitCode=0 Nov 04 01:37:38.253446 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:38.253060 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m2z56" event={"ID":"2a40f207-413e-48d0-b971-f350aae4bbd6","Type":"ContainerDied","Data":"5a63f075d63647f98274aa808935adb4b020ab8606837b99d9af9bfd75272d2b"} Nov 04 01:37:39.398318 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:39.398290 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m2z56" Nov 04 01:37:39.468349 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:39.468320 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a40f207-413e-48d0-b971-f350aae4bbd6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a40f207-413e-48d0-b971-f350aae4bbd6-maas-api-xks-ca\") pod \"2a40f207-413e-48d0-b971-f350aae4bbd6\" (UID: \"2a40f207-413e-48d0-b971-f350aae4bbd6\") " Nov 04 01:37:39.468481 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:39.468445 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a40f207-413e-48d0-b971-f350aae4bbd6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a40f207-413e-48d0-b971-f350aae4bbd6-maas-api-openshift-ca\") pod \"2a40f207-413e-48d0-b971-f350aae4bbd6\" (UID: \"2a40f207-413e-48d0-b971-f350aae4bbd6\") " Nov 04 01:37:39.468537 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:39.468478 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a40f207-413e-48d0-b971-f350aae4bbd6-kube-api-access-kmbf2\" (UniqueName: \"kubernetes.io/projected/2a40f207-413e-48d0-b971-f350aae4bbd6-kube-api-access-kmbf2\") pod \"2a40f207-413e-48d0-b971-f350aae4bbd6\" (UID: \"2a40f207-413e-48d0-b971-f350aae4bbd6\") " Nov 04 01:37:39.468790 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:39.468760 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a40f207-413e-48d0-b971-f350aae4bbd6-maas-api-openshift-ca" pod "2a40f207-413e-48d0-b971-f350aae4bbd6" (UID: "2a40f207-413e-48d0-b971-f350aae4bbd6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:37:39.470540 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:39.470519 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a40f207-413e-48d0-b971-f350aae4bbd6-maas-api-xks-ca" pod "2a40f207-413e-48d0-b971-f350aae4bbd6" (UID: "2a40f207-413e-48d0-b971-f350aae4bbd6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:37:39.470741 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:39.470717 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a40f207-413e-48d0-b971-f350aae4bbd6-kube-api-access-kmbf2" pod "2a40f207-413e-48d0-b971-f350aae4bbd6" (UID: "2a40f207-413e-48d0-b971-f350aae4bbd6"). InnerVolumeSpecName "kube-api-access-kmbf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:37:39.569419 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:39.569358 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a40f207-413e-48d0-b971-f350aae4bbd6-maas-api-openshift-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:39.569419 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:39.569382 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmbf2\" (UniqueName: \"kubernetes.io/projected/2a40f207-413e-48d0-b971-f350aae4bbd6-kube-api-access-kmbf2\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:39.569419 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:39.569393 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a40f207-413e-48d0-b971-f350aae4bbd6-maas-api-xks-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:40.264596 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:40.264566 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m2z56" Nov 04 01:37:40.264775 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:40.264558 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m2z56" event={"ID":"2a40f207-413e-48d0-b971-f350aae4bbd6","Type":"ContainerDied","Data":"cfa4dce6bf5be994c3b81226a942cf3fcf14e82c92a10e7320a9c70b2a124306"} Nov 04 01:37:40.264775 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:40.264696 7059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfa4dce6bf5be994c3b81226a942cf3fcf14e82c92a10e7320a9c70b2a124306" Nov 04 01:37:47.228306 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:47.228227 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jfz9c"] Nov 04 01:37:47.228709 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:47.228693 7059 cpu_manager.go:414] "containerMap: removing container" podUID="2a40f207-413e-48d0-b971-f350aae4bbd6" containerName="revoke-keys" Nov 04 01:37:47.228773 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:47.228710 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a40f207-413e-48d0-b971-f350aae4bbd6" containerName="revoke-keys" Nov 04 01:37:47.228822 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:47.228784 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a40f207-413e-48d0-b971-f350aae4bbd6" containerName="revoke-keys" Nov 04 01:37:47.232307 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:47.232285 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jfz9c" Nov 04 01:37:47.237417 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:47.237399 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:37:47.245684 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:47.245661 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jfz9c"] Nov 04 01:37:47.336568 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:47.336535 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:37:47.343032 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:47.343001 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3ef6ac-14cf-4cd6-a2de-9d241e055e71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-jfz9c\" (UID: \"5c3ef6ac-14cf-4cd6-a2de-9d241e055e71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jfz9c" Nov 04 01:37:47.343145 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:47.343043 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3ef6ac-14cf-4cd6-a2de-9d241e055e71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-jfz9c\" (UID: \"5c3ef6ac-14cf-4cd6-a2de-9d241e055e71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jfz9c" Nov 04 01:37:47.343182 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:47.343138 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r42jh\" (UniqueName: \"kubernetes.io/projected/5c3ef6ac-14cf-4cd6-a2de-9d241e055e71-kube-api-access-r42jh\") pod \"maas-api-revoke-sub-ec843018-jfz9c\" (UID: \"5c3ef6ac-14cf-4cd6-a2de-9d241e055e71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jfz9c" Nov 04 01:37:47.443947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:47.443895 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3ef6ac-14cf-4cd6-a2de-9d241e055e71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-jfz9c\" (UID: \"5c3ef6ac-14cf-4cd6-a2de-9d241e055e71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jfz9c" Nov 04 01:37:47.444101 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:47.443978 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3ef6ac-14cf-4cd6-a2de-9d241e055e71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-jfz9c\" (UID: \"5c3ef6ac-14cf-4cd6-a2de-9d241e055e71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jfz9c" Nov 04 01:37:47.444101 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:47.444027 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r42jh\" (UniqueName: \"kubernetes.io/projected/5c3ef6ac-14cf-4cd6-a2de-9d241e055e71-kube-api-access-r42jh\") pod \"maas-api-revoke-sub-ec843018-jfz9c\" (UID: \"5c3ef6ac-14cf-4cd6-a2de-9d241e055e71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jfz9c" Nov 04 01:37:47.444704 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:47.444635 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3ef6ac-14cf-4cd6-a2de-9d241e055e71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-jfz9c\" (UID: \"5c3ef6ac-14cf-4cd6-a2de-9d241e055e71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jfz9c" Nov 04 01:37:47.446455 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:47.446429 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3ef6ac-14cf-4cd6-a2de-9d241e055e71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-jfz9c\" (UID: \"5c3ef6ac-14cf-4cd6-a2de-9d241e055e71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jfz9c" Nov 04 01:37:47.453418 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:47.453392 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r42jh\" (UniqueName: \"kubernetes.io/projected/5c3ef6ac-14cf-4cd6-a2de-9d241e055e71-kube-api-access-r42jh\") pod \"maas-api-revoke-sub-ec843018-jfz9c\" (UID: \"5c3ef6ac-14cf-4cd6-a2de-9d241e055e71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jfz9c" Nov 04 01:37:47.547612 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:47.547532 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jfz9c" Nov 04 01:37:47.833396 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:47.833368 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jfz9c"] Nov 04 01:37:47.834584 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:37:47.834558 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c3ef6ac_14cf_4cd6_a2de_9d241e055e71.slice/crio-23dbd359126c54ee784a93a25e9263d82bf16a6f4fdf277d804ac6160d85cc47 WatchSource:0}: Error finding container 23dbd359126c54ee784a93a25e9263d82bf16a6f4fdf277d804ac6160d85cc47: Status 404 returned error can't find the container with id 23dbd359126c54ee784a93a25e9263d82bf16a6f4fdf277d804ac6160d85cc47 Nov 04 01:37:48.310209 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:48.310164 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jfz9c" event={"ID":"5c3ef6ac-14cf-4cd6-a2de-9d241e055e71","Type":"ContainerStarted","Data":"20ae6504cabb61f6bbc686071c7e11c7f3482113f38d364e62e913a219af144c"} Nov 04 01:37:48.310745 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:48.310238 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jfz9c" event={"ID":"5c3ef6ac-14cf-4cd6-a2de-9d241e055e71","Type":"ContainerStarted","Data":"23dbd359126c54ee784a93a25e9263d82bf16a6f4fdf277d804ac6160d85cc47"} Nov 04 01:37:48.329888 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:48.329832 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jfz9c" podStartSLOduration=1.329816093 podStartE2EDuration="1.329816093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:37:48.329258576 +0000 UTC m=+2353.271107481" watchObservedRunningTime="2026-11-04 01:37:48.329816093 +0000 UTC m=+2353.271665042" Nov 04 01:37:49.320779 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:49.320743 7059 generic.go:356] "Generic (PLEG): container finished" podID="5c3ef6ac-14cf-4cd6-a2de-9d241e055e71" containerID="20ae6504cabb61f6bbc686071c7e11c7f3482113f38d364e62e913a219af144c" exitCode=0 Nov 04 01:37:49.321252 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:49.320879 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jfz9c" event={"ID":"5c3ef6ac-14cf-4cd6-a2de-9d241e055e71","Type":"ContainerDied","Data":"20ae6504cabb61f6bbc686071c7e11c7f3482113f38d364e62e913a219af144c"} Nov 04 01:37:50.696424 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:50.696080 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jfz9c" Nov 04 01:37:50.737532 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:50.737505 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c3ef6ac-14cf-4cd6-a2de-9d241e055e71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3ef6ac-14cf-4cd6-a2de-9d241e055e71-maas-api-openshift-ca\") pod \"5c3ef6ac-14cf-4cd6-a2de-9d241e055e71\" (UID: \"5c3ef6ac-14cf-4cd6-a2de-9d241e055e71\") " Nov 04 01:37:50.737673 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:50.737549 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c3ef6ac-14cf-4cd6-a2de-9d241e055e71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3ef6ac-14cf-4cd6-a2de-9d241e055e71-maas-api-xks-ca\") pod \"5c3ef6ac-14cf-4cd6-a2de-9d241e055e71\" (UID: \"5c3ef6ac-14cf-4cd6-a2de-9d241e055e71\") " Nov 04 01:37:50.737673 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:50.737620 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c3ef6ac-14cf-4cd6-a2de-9d241e055e71-kube-api-access-r42jh\" (UniqueName: \"kubernetes.io/projected/5c3ef6ac-14cf-4cd6-a2de-9d241e055e71-kube-api-access-r42jh\") pod \"5c3ef6ac-14cf-4cd6-a2de-9d241e055e71\" (UID: \"5c3ef6ac-14cf-4cd6-a2de-9d241e055e71\") " Nov 04 01:37:50.739087 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:50.739056 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c3ef6ac-14cf-4cd6-a2de-9d241e055e71-maas-api-openshift-ca" pod "5c3ef6ac-14cf-4cd6-a2de-9d241e055e71" (UID: "5c3ef6ac-14cf-4cd6-a2de-9d241e055e71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:37:50.841697 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:50.841658 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3ef6ac-14cf-4cd6-a2de-9d241e055e71-maas-api-openshift-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:50.858219 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:50.858188 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c3ef6ac-14cf-4cd6-a2de-9d241e055e71-maas-api-xks-ca" pod "5c3ef6ac-14cf-4cd6-a2de-9d241e055e71" (UID: "5c3ef6ac-14cf-4cd6-a2de-9d241e055e71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:37:50.887149 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:50.887074 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c3ef6ac-14cf-4cd6-a2de-9d241e055e71-kube-api-access-r42jh" pod "5c3ef6ac-14cf-4cd6-a2de-9d241e055e71" (UID: "5c3ef6ac-14cf-4cd6-a2de-9d241e055e71"). InnerVolumeSpecName "kube-api-access-r42jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:37:50.944721 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:50.944644 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r42jh\" (UniqueName: \"kubernetes.io/projected/5c3ef6ac-14cf-4cd6-a2de-9d241e055e71-kube-api-access-r42jh\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:50.944721 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:50.944686 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3ef6ac-14cf-4cd6-a2de-9d241e055e71-maas-api-xks-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:51.333098 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:51.333067 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jfz9c" Nov 04 01:37:51.333315 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:51.333064 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jfz9c" event={"ID":"5c3ef6ac-14cf-4cd6-a2de-9d241e055e71","Type":"ContainerDied","Data":"23dbd359126c54ee784a93a25e9263d82bf16a6f4fdf277d804ac6160d85cc47"} Nov 04 01:37:51.333315 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:37:51.333193 7059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23dbd359126c54ee784a93a25e9263d82bf16a6f4fdf277d804ac6160d85cc47" Nov 04 01:38:06.479584 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:38:06.479545 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:38:11.582274 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:11.582236 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w2mnf"] Nov 04 01:38:11.582722 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:11.582708 7059 cpu_manager.go:414] "containerMap: removing container" podUID="5c3ef6ac-14cf-4cd6-a2de-9d241e055e71" containerName="revoke-keys" Nov 04 01:38:11.582779 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:11.582723 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c3ef6ac-14cf-4cd6-a2de-9d241e055e71" containerName="revoke-keys" Nov 04 01:38:11.582834 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:11.582824 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c3ef6ac-14cf-4cd6-a2de-9d241e055e71" containerName="revoke-keys" Nov 04 01:38:11.586203 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:11.586183 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w2mnf" Nov 04 01:38:11.592258 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:11.592194 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:38:11.592481 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:11.592460 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:38:11.597405 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:11.597128 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w2mnf"] Nov 04 01:38:11.687358 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:11.687316 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00341182-d49f-499c-a3eb-2c26d7987a35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-w2mnf\" (UID: \"00341182-d49f-499c-a3eb-2c26d7987a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w2mnf" Nov 04 01:38:11.687534 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:11.687467 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00341182-d49f-499c-a3eb-2c26d7987a35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-w2mnf\" (UID: \"00341182-d49f-499c-a3eb-2c26d7987a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w2mnf" Nov 04 01:38:11.687606 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:11.687567 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q42qb\" (UniqueName: \"kubernetes.io/projected/00341182-d49f-499c-a3eb-2c26d7987a35-kube-api-access-q42qb\") pod \"maas-api-revoke-sub-046e1343-w2mnf\" (UID: \"00341182-d49f-499c-a3eb-2c26d7987a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w2mnf" Nov 04 01:38:11.788791 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:11.788755 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q42qb\" (UniqueName: \"kubernetes.io/projected/00341182-d49f-499c-a3eb-2c26d7987a35-kube-api-access-q42qb\") pod \"maas-api-revoke-sub-046e1343-w2mnf\" (UID: \"00341182-d49f-499c-a3eb-2c26d7987a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w2mnf" Nov 04 01:38:11.788989 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:11.788816 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00341182-d49f-499c-a3eb-2c26d7987a35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-w2mnf\" (UID: \"00341182-d49f-499c-a3eb-2c26d7987a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w2mnf" Nov 04 01:38:11.788989 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:11.788880 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00341182-d49f-499c-a3eb-2c26d7987a35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-w2mnf\" (UID: \"00341182-d49f-499c-a3eb-2c26d7987a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w2mnf" Nov 04 01:38:11.789485 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:11.789456 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00341182-d49f-499c-a3eb-2c26d7987a35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-w2mnf\" (UID: \"00341182-d49f-499c-a3eb-2c26d7987a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w2mnf" Nov 04 01:38:11.791348 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:11.791325 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00341182-d49f-499c-a3eb-2c26d7987a35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-w2mnf\" (UID: \"00341182-d49f-499c-a3eb-2c26d7987a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w2mnf" Nov 04 01:38:11.799477 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:11.799457 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q42qb\" (UniqueName: \"kubernetes.io/projected/00341182-d49f-499c-a3eb-2c26d7987a35-kube-api-access-q42qb\") pod \"maas-api-revoke-sub-046e1343-w2mnf\" (UID: \"00341182-d49f-499c-a3eb-2c26d7987a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w2mnf" Nov 04 01:38:11.917602 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:11.917528 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w2mnf" Nov 04 01:38:12.192121 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:12.191532 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w2mnf"] Nov 04 01:38:12.192808 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:38:12.192783 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00341182_d49f_499c_a3eb_2c26d7987a35.slice/crio-ae654f0f47fc42e4a7209ce1524c8131851651045fb498d14bb40228d56fdee5 WatchSource:0}: Error finding container ae654f0f47fc42e4a7209ce1524c8131851651045fb498d14bb40228d56fdee5: Status 404 returned error can't find the container with id ae654f0f47fc42e4a7209ce1524c8131851651045fb498d14bb40228d56fdee5 Nov 04 01:38:12.479818 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:12.479782 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w2mnf" event={"ID":"00341182-d49f-499c-a3eb-2c26d7987a35","Type":"ContainerStarted","Data":"69bb95b913afa05dde926e61a529a4a4d6d5f412313170aabdd4161e53edcaa1"} Nov 04 01:38:12.479962 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:12.479848 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w2mnf" event={"ID":"00341182-d49f-499c-a3eb-2c26d7987a35","Type":"ContainerStarted","Data":"ae654f0f47fc42e4a7209ce1524c8131851651045fb498d14bb40228d56fdee5"} Nov 04 01:38:12.498547 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:12.498505 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w2mnf" podStartSLOduration=1.498491647 podStartE2EDuration="1.498491647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:38:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:38:12.49555456 +0000 UTC m=+2377.437403479" watchObservedRunningTime="2026-11-04 01:38:12.498491647 +0000 UTC m=+2377.440340571" Nov 04 01:38:13.487619 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:13.487587 7059 generic.go:356] "Generic (PLEG): container finished" podID="00341182-d49f-499c-a3eb-2c26d7987a35" containerID="69bb95b913afa05dde926e61a529a4a4d6d5f412313170aabdd4161e53edcaa1" exitCode=0 Nov 04 01:38:13.487993 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:13.487666 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w2mnf" event={"ID":"00341182-d49f-499c-a3eb-2c26d7987a35","Type":"ContainerDied","Data":"69bb95b913afa05dde926e61a529a4a4d6d5f412313170aabdd4161e53edcaa1"} Nov 04 01:38:14.633439 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:14.633413 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w2mnf" Nov 04 01:38:14.714438 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:14.714397 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00341182-d49f-499c-a3eb-2c26d7987a35-kube-api-access-q42qb\" (UniqueName: \"kubernetes.io/projected/00341182-d49f-499c-a3eb-2c26d7987a35-kube-api-access-q42qb\") pod \"00341182-d49f-499c-a3eb-2c26d7987a35\" (UID: \"00341182-d49f-499c-a3eb-2c26d7987a35\") " Nov 04 01:38:14.714593 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:14.714482 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00341182-d49f-499c-a3eb-2c26d7987a35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00341182-d49f-499c-a3eb-2c26d7987a35-maas-api-xks-ca\") pod \"00341182-d49f-499c-a3eb-2c26d7987a35\" (UID: \"00341182-d49f-499c-a3eb-2c26d7987a35\") " Nov 04 01:38:14.714593 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:14.714509 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00341182-d49f-499c-a3eb-2c26d7987a35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00341182-d49f-499c-a3eb-2c26d7987a35-maas-api-openshift-ca\") pod \"00341182-d49f-499c-a3eb-2c26d7987a35\" (UID: \"00341182-d49f-499c-a3eb-2c26d7987a35\") " Nov 04 01:38:14.714910 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:14.714886 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00341182-d49f-499c-a3eb-2c26d7987a35-maas-api-openshift-ca" pod "00341182-d49f-499c-a3eb-2c26d7987a35" (UID: "00341182-d49f-499c-a3eb-2c26d7987a35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:38:14.716598 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:14.716575 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00341182-d49f-499c-a3eb-2c26d7987a35-kube-api-access-q42qb" pod "00341182-d49f-499c-a3eb-2c26d7987a35" (UID: "00341182-d49f-499c-a3eb-2c26d7987a35"). InnerVolumeSpecName "kube-api-access-q42qb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:38:14.716740 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:14.716715 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00341182-d49f-499c-a3eb-2c26d7987a35-maas-api-xks-ca" pod "00341182-d49f-499c-a3eb-2c26d7987a35" (UID: "00341182-d49f-499c-a3eb-2c26d7987a35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:38:14.816262 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:14.816207 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q42qb\" (UniqueName: \"kubernetes.io/projected/00341182-d49f-499c-a3eb-2c26d7987a35-kube-api-access-q42qb\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:14.816262 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:14.816229 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00341182-d49f-499c-a3eb-2c26d7987a35-maas-api-xks-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:14.816262 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:14.816238 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00341182-d49f-499c-a3eb-2c26d7987a35-maas-api-openshift-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:15.500221 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:15.500122 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w2mnf" event={"ID":"00341182-d49f-499c-a3eb-2c26d7987a35","Type":"ContainerDied","Data":"ae654f0f47fc42e4a7209ce1524c8131851651045fb498d14bb40228d56fdee5"} Nov 04 01:38:15.500221 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:15.500156 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w2mnf" Nov 04 01:38:15.500221 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:15.500172 7059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae654f0f47fc42e4a7209ce1524c8131851651045fb498d14bb40228d56fdee5" Nov 04 01:38:21.803331 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:21.803293 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sd6ps"] Nov 04 01:38:21.803789 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:21.803749 7059 cpu_manager.go:414] "containerMap: removing container" podUID="00341182-d49f-499c-a3eb-2c26d7987a35" containerName="revoke-keys" Nov 04 01:38:21.803789 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:21.803761 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00341182-d49f-499c-a3eb-2c26d7987a35" containerName="revoke-keys" Nov 04 01:38:21.803861 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:21.803855 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="00341182-d49f-499c-a3eb-2c26d7987a35" containerName="revoke-keys" Nov 04 01:38:21.807506 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:21.807485 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sd6ps" Nov 04 01:38:21.813401 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:21.813373 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:38:21.813541 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:21.813489 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:38:21.817768 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:21.816448 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sd6ps"] Nov 04 01:38:21.879129 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:21.879097 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cee30f6-a675-4775-b767-b78376d9257c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-sd6ps\" (UID: \"5cee30f6-a675-4775-b767-b78376d9257c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sd6ps" Nov 04 01:38:21.879299 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:21.879154 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cee30f6-a675-4775-b767-b78376d9257c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-sd6ps\" (UID: \"5cee30f6-a675-4775-b767-b78376d9257c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sd6ps" Nov 04 01:38:21.879299 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:21.879219 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92sr5\" (UniqueName: \"kubernetes.io/projected/5cee30f6-a675-4775-b767-b78376d9257c-kube-api-access-92sr5\") pod \"maas-api-revoke-sub-d3fba056-sd6ps\" (UID: \"5cee30f6-a675-4775-b767-b78376d9257c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sd6ps" Nov 04 01:38:21.980445 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:21.980411 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92sr5\" (UniqueName: \"kubernetes.io/projected/5cee30f6-a675-4775-b767-b78376d9257c-kube-api-access-92sr5\") pod \"maas-api-revoke-sub-d3fba056-sd6ps\" (UID: \"5cee30f6-a675-4775-b767-b78376d9257c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sd6ps" Nov 04 01:38:21.980595 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:21.980525 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cee30f6-a675-4775-b767-b78376d9257c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-sd6ps\" (UID: \"5cee30f6-a675-4775-b767-b78376d9257c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sd6ps" Nov 04 01:38:21.980595 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:21.980554 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cee30f6-a675-4775-b767-b78376d9257c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-sd6ps\" (UID: \"5cee30f6-a675-4775-b767-b78376d9257c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sd6ps" Nov 04 01:38:21.981249 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:21.981224 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cee30f6-a675-4775-b767-b78376d9257c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-sd6ps\" (UID: \"5cee30f6-a675-4775-b767-b78376d9257c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sd6ps" Nov 04 01:38:21.983011 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:21.982979 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cee30f6-a675-4775-b767-b78376d9257c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-sd6ps\" (UID: \"5cee30f6-a675-4775-b767-b78376d9257c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sd6ps" Nov 04 01:38:21.988402 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:21.988379 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92sr5\" (UniqueName: \"kubernetes.io/projected/5cee30f6-a675-4775-b767-b78376d9257c-kube-api-access-92sr5\") pod \"maas-api-revoke-sub-d3fba056-sd6ps\" (UID: \"5cee30f6-a675-4775-b767-b78376d9257c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sd6ps" Nov 04 01:38:22.135284 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:22.135216 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sd6ps" Nov 04 01:38:22.411315 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:22.411288 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sd6ps"] Nov 04 01:38:22.412839 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:38:22.412809 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cee30f6_a675_4775_b767_b78376d9257c.slice/crio-a2346475273017024deed80ff1b7d056d8d8a143a68373bd3c06de2afaa14712 WatchSource:0}: Error finding container a2346475273017024deed80ff1b7d056d8d8a143a68373bd3c06de2afaa14712: Status 404 returned error can't find the container with id a2346475273017024deed80ff1b7d056d8d8a143a68373bd3c06de2afaa14712 Nov 04 01:38:22.540160 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:22.540122 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sd6ps" event={"ID":"5cee30f6-a675-4775-b767-b78376d9257c","Type":"ContainerStarted","Data":"a2346475273017024deed80ff1b7d056d8d8a143a68373bd3c06de2afaa14712"} Nov 04 01:38:23.550102 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:23.550000 7059 generic.go:356] "Generic (PLEG): container finished" podID="5cee30f6-a675-4775-b767-b78376d9257c" containerID="01caf3df7a4de61e753971f360d0363ef02da43b264e84653604fe670165b686" exitCode=0 Nov 04 01:38:23.550488 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:23.550105 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sd6ps" event={"ID":"5cee30f6-a675-4775-b767-b78376d9257c","Type":"ContainerDied","Data":"01caf3df7a4de61e753971f360d0363ef02da43b264e84653604fe670165b686"} Nov 04 01:38:25.047946 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:25.038293 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sd6ps" Nov 04 01:38:25.125623 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:25.125590 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cee30f6-a675-4775-b767-b78376d9257c-kube-api-access-92sr5\" (UniqueName: \"kubernetes.io/projected/5cee30f6-a675-4775-b767-b78376d9257c-kube-api-access-92sr5\") pod \"5cee30f6-a675-4775-b767-b78376d9257c\" (UID: \"5cee30f6-a675-4775-b767-b78376d9257c\") " Nov 04 01:38:25.125832 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:25.125644 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cee30f6-a675-4775-b767-b78376d9257c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cee30f6-a675-4775-b767-b78376d9257c-maas-api-xks-ca\") pod \"5cee30f6-a675-4775-b767-b78376d9257c\" (UID: \"5cee30f6-a675-4775-b767-b78376d9257c\") " Nov 04 01:38:25.126292 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:25.126137 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cee30f6-a675-4775-b767-b78376d9257c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cee30f6-a675-4775-b767-b78376d9257c-maas-api-openshift-ca\") pod \"5cee30f6-a675-4775-b767-b78376d9257c\" (UID: \"5cee30f6-a675-4775-b767-b78376d9257c\") " Nov 04 01:38:25.126614 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:25.126584 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cee30f6-a675-4775-b767-b78376d9257c-maas-api-openshift-ca" pod "5cee30f6-a675-4775-b767-b78376d9257c" (UID: "5cee30f6-a675-4775-b767-b78376d9257c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:38:25.126817 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:25.126684 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cee30f6-a675-4775-b767-b78376d9257c-maas-api-openshift-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:25.128081 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:25.128055 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cee30f6-a675-4775-b767-b78376d9257c-kube-api-access-92sr5" pod "5cee30f6-a675-4775-b767-b78376d9257c" (UID: "5cee30f6-a675-4775-b767-b78376d9257c"). InnerVolumeSpecName "kube-api-access-92sr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:38:25.128177 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:25.128153 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cee30f6-a675-4775-b767-b78376d9257c-maas-api-xks-ca" pod "5cee30f6-a675-4775-b767-b78376d9257c" (UID: "5cee30f6-a675-4775-b767-b78376d9257c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:38:25.227621 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:25.227543 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92sr5\" (UniqueName: \"kubernetes.io/projected/5cee30f6-a675-4775-b767-b78376d9257c-kube-api-access-92sr5\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:25.227621 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:25.227572 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cee30f6-a675-4775-b767-b78376d9257c-maas-api-xks-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:25.588006 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:25.587499 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sd6ps" event={"ID":"5cee30f6-a675-4775-b767-b78376d9257c","Type":"ContainerDied","Data":"a2346475273017024deed80ff1b7d056d8d8a143a68373bd3c06de2afaa14712"} Nov 04 01:38:25.588006 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:25.587566 7059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2346475273017024deed80ff1b7d056d8d8a143a68373bd3c06de2afaa14712" Nov 04 01:38:25.588006 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:25.587641 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sd6ps" Nov 04 01:38:36.530760 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:38:36.530721 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:38:42.642024 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:42.641985 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-x9p7x"] Nov 04 01:38:42.642540 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:42.642482 7059 cpu_manager.go:414] "containerMap: removing container" podUID="5cee30f6-a675-4775-b767-b78376d9257c" containerName="revoke-keys" Nov 04 01:38:42.642540 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:42.642496 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cee30f6-a675-4775-b767-b78376d9257c" containerName="revoke-keys" Nov 04 01:38:42.642658 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:42.642606 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cee30f6-a675-4775-b767-b78376d9257c" containerName="revoke-keys" Nov 04 01:38:42.645690 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:42.645674 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-x9p7x" Nov 04 01:38:42.654218 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:42.654035 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:38:42.654371 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:42.654233 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:38:42.659244 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:42.659219 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-x9p7x"] Nov 04 01:38:42.735549 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:42.735518 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmj7v\" (UniqueName: \"kubernetes.io/projected/58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb-kube-api-access-zmj7v\") pod \"maas-api-revoke-sub-ae594979-x9p7x\" (UID: \"58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-x9p7x" Nov 04 01:38:42.735701 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:42.735556 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-x9p7x\" (UID: \"58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-x9p7x" Nov 04 01:38:42.735701 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:42.735678 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-x9p7x\" (UID: \"58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-x9p7x" Nov 04 01:38:42.836358 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:42.836323 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmj7v\" (UniqueName: \"kubernetes.io/projected/58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb-kube-api-access-zmj7v\") pod \"maas-api-revoke-sub-ae594979-x9p7x\" (UID: \"58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-x9p7x" Nov 04 01:38:42.836527 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:42.836378 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-x9p7x\" (UID: \"58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-x9p7x" Nov 04 01:38:42.836527 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:42.836479 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-x9p7x\" (UID: \"58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-x9p7x" Nov 04 01:38:42.837261 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:42.837180 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-x9p7x\" (UID: \"58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-x9p7x" Nov 04 01:38:42.839022 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:42.838999 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-x9p7x\" (UID: \"58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-x9p7x" Nov 04 01:38:42.846421 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:42.846396 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmj7v\" (UniqueName: \"kubernetes.io/projected/58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb-kube-api-access-zmj7v\") pod \"maas-api-revoke-sub-ae594979-x9p7x\" (UID: \"58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-x9p7x" Nov 04 01:38:42.963180 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:42.963158 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-x9p7x" Nov 04 01:38:43.242161 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:43.242134 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-x9p7x"] Nov 04 01:38:43.245234 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:38:43.245210 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58c5ef4a_83ee_4b36_8b52_bfe84e7a40eb.slice/crio-5de97c40eb2158945cd37f5978a57896782d464f1e00732e756e4444306f5e29 WatchSource:0}: Error finding container 5de97c40eb2158945cd37f5978a57896782d464f1e00732e756e4444306f5e29: Status 404 returned error can't find the container with id 5de97c40eb2158945cd37f5978a57896782d464f1e00732e756e4444306f5e29 Nov 04 01:38:43.739602 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:43.739559 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-x9p7x" event={"ID":"58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb","Type":"ContainerStarted","Data":"8246577fc36aac14284e4e6c99742f4edf621569fdd940bd0a9804e1503334a4"} Nov 04 01:38:43.739983 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:43.739650 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-x9p7x" event={"ID":"58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb","Type":"ContainerStarted","Data":"5de97c40eb2158945cd37f5978a57896782d464f1e00732e756e4444306f5e29"} Nov 04 01:38:43.759964 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:43.759892 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-x9p7x" podStartSLOduration=1.75988074 podStartE2EDuration="1.75988074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:38:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:38:43.758109875 +0000 UTC m=+2408.699958780" watchObservedRunningTime="2026-11-04 01:38:43.75988074 +0000 UTC m=+2408.701729643" Nov 04 01:38:44.747034 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:44.747001 7059 generic.go:356] "Generic (PLEG): container finished" podID="58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb" containerID="8246577fc36aac14284e4e6c99742f4edf621569fdd940bd0a9804e1503334a4" exitCode=0 Nov 04 01:38:44.747417 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:44.747049 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-x9p7x" event={"ID":"58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb","Type":"ContainerDied","Data":"8246577fc36aac14284e4e6c99742f4edf621569fdd940bd0a9804e1503334a4"} Nov 04 01:38:45.892640 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:45.892617 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-x9p7x" Nov 04 01:38:46.068075 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:46.068003 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb-maas-api-xks-ca\") pod \"58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb\" (UID: \"58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb\") " Nov 04 01:38:46.068214 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:46.068105 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb-kube-api-access-zmj7v\" (UniqueName: \"kubernetes.io/projected/58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb-kube-api-access-zmj7v\") pod \"58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb\" (UID: \"58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb\") " Nov 04 01:38:46.068214 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:46.068141 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb-maas-api-openshift-ca\") pod \"58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb\" (UID: \"58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb\") " Nov 04 01:38:46.068565 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:46.068539 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb-maas-api-openshift-ca" pod "58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb" (UID: "58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:38:46.070205 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:46.070182 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb-maas-api-xks-ca" pod "58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb" (UID: "58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:38:46.070255 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:46.070235 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb-kube-api-access-zmj7v" pod "58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb" (UID: "58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb"). InnerVolumeSpecName "kube-api-access-zmj7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:38:46.169146 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:46.169118 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb-maas-api-xks-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:46.169146 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:46.169143 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmj7v\" (UniqueName: \"kubernetes.io/projected/58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb-kube-api-access-zmj7v\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:46.169321 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:46.169153 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb-maas-api-openshift-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:46.758840 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:46.758805 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-x9p7x" event={"ID":"58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb","Type":"ContainerDied","Data":"5de97c40eb2158945cd37f5978a57896782d464f1e00732e756e4444306f5e29"} Nov 04 01:38:46.759033 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:46.758859 7059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5de97c40eb2158945cd37f5978a57896782d464f1e00732e756e4444306f5e29" Nov 04 01:38:46.759033 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:38:46.758814 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-x9p7x" Nov 04 01:39:06.479410 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:39:06.479373 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:39:15.570065 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:15.569977 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xwddf"] Nov 04 01:39:15.571535 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:15.571507 7059 cpu_manager.go:414] "containerMap: removing container" podUID="58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb" containerName="revoke-keys" Nov 04 01:39:15.571704 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:15.571687 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb" containerName="revoke-keys" Nov 04 01:39:15.571961 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:15.571945 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb" containerName="revoke-keys" Nov 04 01:39:15.584116 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:15.584097 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xwddf"] Nov 04 01:39:15.584379 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:15.584315 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xwddf" Nov 04 01:39:15.594947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:15.593673 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:39:15.594947 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:15.593754 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:39:15.607730 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:15.607704 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhfd9\" (UniqueName: \"kubernetes.io/projected/6620248b-44a6-40f4-915f-28721500f1f5-kube-api-access-bhfd9\") pod \"maas-api-revoke-sub-79805ca9-xwddf\" (UID: \"6620248b-44a6-40f4-915f-28721500f1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xwddf" Nov 04 01:39:15.607963 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:15.607942 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6620248b-44a6-40f4-915f-28721500f1f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-xwddf\" (UID: \"6620248b-44a6-40f4-915f-28721500f1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xwddf" Nov 04 01:39:15.608175 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:15.608158 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6620248b-44a6-40f4-915f-28721500f1f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-xwddf\" (UID: \"6620248b-44a6-40f4-915f-28721500f1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xwddf" Nov 04 01:39:15.710322 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:15.709882 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhfd9\" (UniqueName: \"kubernetes.io/projected/6620248b-44a6-40f4-915f-28721500f1f5-kube-api-access-bhfd9\") pod \"maas-api-revoke-sub-79805ca9-xwddf\" (UID: \"6620248b-44a6-40f4-915f-28721500f1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xwddf" Nov 04 01:39:15.710715 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:15.710692 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6620248b-44a6-40f4-915f-28721500f1f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-xwddf\" (UID: \"6620248b-44a6-40f4-915f-28721500f1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xwddf" Nov 04 01:39:15.711537 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:15.711505 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6620248b-44a6-40f4-915f-28721500f1f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-xwddf\" (UID: \"6620248b-44a6-40f4-915f-28721500f1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xwddf" Nov 04 01:39:15.712671 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:15.712552 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6620248b-44a6-40f4-915f-28721500f1f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-xwddf\" (UID: \"6620248b-44a6-40f4-915f-28721500f1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xwddf" Nov 04 01:39:15.716275 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:15.716229 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6620248b-44a6-40f4-915f-28721500f1f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-xwddf\" (UID: \"6620248b-44a6-40f4-915f-28721500f1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xwddf" Nov 04 01:39:15.722560 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:15.722516 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhfd9\" (UniqueName: \"kubernetes.io/projected/6620248b-44a6-40f4-915f-28721500f1f5-kube-api-access-bhfd9\") pod \"maas-api-revoke-sub-79805ca9-xwddf\" (UID: \"6620248b-44a6-40f4-915f-28721500f1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xwddf" Nov 04 01:39:15.927186 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:15.927156 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xwddf" Nov 04 01:39:16.345952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:16.345908 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xwddf"] Nov 04 01:39:16.346696 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:39:16.346670 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6620248b_44a6_40f4_915f_28721500f1f5.slice/crio-42a4612ab83eb198f57b7808e17958175d6c62566d55581b2443c1c9a51785ba WatchSource:0}: Error finding container 42a4612ab83eb198f57b7808e17958175d6c62566d55581b2443c1c9a51785ba: Status 404 returned error can't find the container with id 42a4612ab83eb198f57b7808e17958175d6c62566d55581b2443c1c9a51785ba Nov 04 01:39:17.019556 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:17.019514 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xwddf" event={"ID":"6620248b-44a6-40f4-915f-28721500f1f5","Type":"ContainerStarted","Data":"4a81935cd70a9cf69eeee51c5bac1d5b280b74b61b9f2469e82ab1187817da6e"} Nov 04 01:39:17.020019 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:17.019589 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xwddf" event={"ID":"6620248b-44a6-40f4-915f-28721500f1f5","Type":"ContainerStarted","Data":"42a4612ab83eb198f57b7808e17958175d6c62566d55581b2443c1c9a51785ba"} Nov 04 01:39:18.030724 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:18.030690 7059 generic.go:356] "Generic (PLEG): container finished" podID="6620248b-44a6-40f4-915f-28721500f1f5" containerID="4a81935cd70a9cf69eeee51c5bac1d5b280b74b61b9f2469e82ab1187817da6e" exitCode=0 Nov 04 01:39:18.031233 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:18.030725 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xwddf" event={"ID":"6620248b-44a6-40f4-915f-28721500f1f5","Type":"ContainerDied","Data":"4a81935cd70a9cf69eeee51c5bac1d5b280b74b61b9f2469e82ab1187817da6e"} Nov 04 01:39:19.183692 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:19.183666 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xwddf" Nov 04 01:39:19.252224 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:19.252192 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6620248b-44a6-40f4-915f-28721500f1f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6620248b-44a6-40f4-915f-28721500f1f5-maas-api-xks-ca\") pod \"6620248b-44a6-40f4-915f-28721500f1f5\" (UID: \"6620248b-44a6-40f4-915f-28721500f1f5\") " Nov 04 01:39:19.252362 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:19.252265 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6620248b-44a6-40f4-915f-28721500f1f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6620248b-44a6-40f4-915f-28721500f1f5-maas-api-openshift-ca\") pod \"6620248b-44a6-40f4-915f-28721500f1f5\" (UID: \"6620248b-44a6-40f4-915f-28721500f1f5\") " Nov 04 01:39:19.252362 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:19.252342 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6620248b-44a6-40f4-915f-28721500f1f5-kube-api-access-bhfd9\" (UniqueName: \"kubernetes.io/projected/6620248b-44a6-40f4-915f-28721500f1f5-kube-api-access-bhfd9\") pod \"6620248b-44a6-40f4-915f-28721500f1f5\" (UID: \"6620248b-44a6-40f4-915f-28721500f1f5\") " Nov 04 01:39:19.252606 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:19.252570 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6620248b-44a6-40f4-915f-28721500f1f5-maas-api-openshift-ca" pod "6620248b-44a6-40f4-915f-28721500f1f5" (UID: "6620248b-44a6-40f4-915f-28721500f1f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:39:19.253068 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:19.252828 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6620248b-44a6-40f4-915f-28721500f1f5-maas-api-openshift-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:19.254500 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:19.254473 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6620248b-44a6-40f4-915f-28721500f1f5-maas-api-xks-ca" pod "6620248b-44a6-40f4-915f-28721500f1f5" (UID: "6620248b-44a6-40f4-915f-28721500f1f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:39:19.254603 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:19.254527 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6620248b-44a6-40f4-915f-28721500f1f5-kube-api-access-bhfd9" pod "6620248b-44a6-40f4-915f-28721500f1f5" (UID: "6620248b-44a6-40f4-915f-28721500f1f5"). InnerVolumeSpecName "kube-api-access-bhfd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:39:19.353503 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:19.353445 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6620248b-44a6-40f4-915f-28721500f1f5-maas-api-xks-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:19.353503 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:19.353466 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhfd9\" (UniqueName: \"kubernetes.io/projected/6620248b-44a6-40f4-915f-28721500f1f5-kube-api-access-bhfd9\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:20.044712 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:20.044683 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xwddf" Nov 04 01:39:20.044905 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:20.044709 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xwddf" event={"ID":"6620248b-44a6-40f4-915f-28721500f1f5","Type":"ContainerDied","Data":"42a4612ab83eb198f57b7808e17958175d6c62566d55581b2443c1c9a51785ba"} Nov 04 01:39:20.044905 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:20.044757 7059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42a4612ab83eb198f57b7808e17958175d6c62566d55581b2443c1c9a51785ba" Nov 04 01:39:30.152009 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:30.151975 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zfmjd"] Nov 04 01:39:30.153433 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:30.153408 7059 cpu_manager.go:414] "containerMap: removing container" podUID="6620248b-44a6-40f4-915f-28721500f1f5" containerName="revoke-keys" Nov 04 01:39:30.153615 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:30.153597 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6620248b-44a6-40f4-915f-28721500f1f5" containerName="revoke-keys" Nov 04 01:39:30.164157 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:30.164126 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="6620248b-44a6-40f4-915f-28721500f1f5" containerName="revoke-keys" Nov 04 01:39:30.219853 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:30.219825 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zfmjd"] Nov 04 01:39:30.221162 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:30.219979 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zfmjd" Nov 04 01:39:30.226572 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:30.226549 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:39:30.228778 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:30.228758 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:39:30.409319 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:30.409216 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13704caf-55c8-4c4c-985f-da9755654491-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-zfmjd\" (UID: \"13704caf-55c8-4c4c-985f-da9755654491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zfmjd" Nov 04 01:39:30.409642 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:30.409618 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13704caf-55c8-4c4c-985f-da9755654491-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-zfmjd\" (UID: \"13704caf-55c8-4c4c-985f-da9755654491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zfmjd" Nov 04 01:39:30.409863 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:30.409843 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d58k4\" (UniqueName: \"kubernetes.io/projected/13704caf-55c8-4c4c-985f-da9755654491-kube-api-access-d58k4\") pod \"maas-api-revoke-sub-0b7e8edb-zfmjd\" (UID: \"13704caf-55c8-4c4c-985f-da9755654491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zfmjd" Nov 04 01:39:30.511691 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:30.511660 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13704caf-55c8-4c4c-985f-da9755654491-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-zfmjd\" (UID: \"13704caf-55c8-4c4c-985f-da9755654491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zfmjd" Nov 04 01:39:30.512022 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:30.511985 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13704caf-55c8-4c4c-985f-da9755654491-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-zfmjd\" (UID: \"13704caf-55c8-4c4c-985f-da9755654491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zfmjd" Nov 04 01:39:30.512231 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:30.512209 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d58k4\" (UniqueName: \"kubernetes.io/projected/13704caf-55c8-4c4c-985f-da9755654491-kube-api-access-d58k4\") pod \"maas-api-revoke-sub-0b7e8edb-zfmjd\" (UID: \"13704caf-55c8-4c4c-985f-da9755654491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zfmjd" Nov 04 01:39:30.514005 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:30.513982 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13704caf-55c8-4c4c-985f-da9755654491-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-zfmjd\" (UID: \"13704caf-55c8-4c4c-985f-da9755654491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zfmjd" Nov 04 01:39:30.517635 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:30.517615 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13704caf-55c8-4c4c-985f-da9755654491-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-zfmjd\" (UID: \"13704caf-55c8-4c4c-985f-da9755654491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zfmjd" Nov 04 01:39:30.520698 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:30.520673 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d58k4\" (UniqueName: \"kubernetes.io/projected/13704caf-55c8-4c4c-985f-da9755654491-kube-api-access-d58k4\") pod \"maas-api-revoke-sub-0b7e8edb-zfmjd\" (UID: \"13704caf-55c8-4c4c-985f-da9755654491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zfmjd" Nov 04 01:39:30.552948 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:30.551047 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zfmjd" Nov 04 01:39:30.975986 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:30.973695 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zfmjd"] Nov 04 01:39:31.124035 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:31.124002 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zfmjd" event={"ID":"13704caf-55c8-4c4c-985f-da9755654491","Type":"ContainerStarted","Data":"acee6a86a515c4346bc199cbc0383d0f1e4398beb9d3c668022a4144ddfc2e68"} Nov 04 01:39:32.131013 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:32.130978 7059 generic.go:356] "Generic (PLEG): container finished" podID="13704caf-55c8-4c4c-985f-da9755654491" containerID="23c0a0ff7a1f9e1a021752b72b9bd93061973746c395b357d34b271ee959c987" exitCode=0 Nov 04 01:39:32.131468 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:32.131069 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zfmjd" event={"ID":"13704caf-55c8-4c4c-985f-da9755654491","Type":"ContainerDied","Data":"23c0a0ff7a1f9e1a021752b72b9bd93061973746c395b357d34b271ee959c987"} Nov 04 01:39:33.283007 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:33.282979 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zfmjd" Nov 04 01:39:33.344256 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:33.344228 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13704caf-55c8-4c4c-985f-da9755654491-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13704caf-55c8-4c4c-985f-da9755654491-maas-api-openshift-ca\") pod \"13704caf-55c8-4c4c-985f-da9755654491\" (UID: \"13704caf-55c8-4c4c-985f-da9755654491\") " Nov 04 01:39:33.344432 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:33.344266 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13704caf-55c8-4c4c-985f-da9755654491-kube-api-access-d58k4\" (UniqueName: \"kubernetes.io/projected/13704caf-55c8-4c4c-985f-da9755654491-kube-api-access-d58k4\") pod \"13704caf-55c8-4c4c-985f-da9755654491\" (UID: \"13704caf-55c8-4c4c-985f-da9755654491\") " Nov 04 01:39:33.344432 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:33.344372 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13704caf-55c8-4c4c-985f-da9755654491-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13704caf-55c8-4c4c-985f-da9755654491-maas-api-xks-ca\") pod \"13704caf-55c8-4c4c-985f-da9755654491\" (UID: \"13704caf-55c8-4c4c-985f-da9755654491\") " Nov 04 01:39:33.344569 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:33.344548 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13704caf-55c8-4c4c-985f-da9755654491-maas-api-openshift-ca" pod "13704caf-55c8-4c4c-985f-da9755654491" (UID: "13704caf-55c8-4c4c-985f-da9755654491"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:39:33.344725 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:33.344704 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13704caf-55c8-4c4c-985f-da9755654491-maas-api-openshift-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:33.346512 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:33.346488 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13704caf-55c8-4c4c-985f-da9755654491-maas-api-xks-ca" pod "13704caf-55c8-4c4c-985f-da9755654491" (UID: "13704caf-55c8-4c4c-985f-da9755654491"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:39:33.346566 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:33.346492 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13704caf-55c8-4c4c-985f-da9755654491-kube-api-access-d58k4" pod "13704caf-55c8-4c4c-985f-da9755654491" (UID: "13704caf-55c8-4c4c-985f-da9755654491"). InnerVolumeSpecName "kube-api-access-d58k4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:39:33.445189 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:33.445164 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d58k4\" (UniqueName: \"kubernetes.io/projected/13704caf-55c8-4c4c-985f-da9755654491-kube-api-access-d58k4\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:33.445189 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:33.445189 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13704caf-55c8-4c4c-985f-da9755654491-maas-api-xks-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:34.145067 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:34.145030 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zfmjd" Nov 04 01:39:34.145067 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:34.145059 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zfmjd" event={"ID":"13704caf-55c8-4c4c-985f-da9755654491","Type":"ContainerDied","Data":"acee6a86a515c4346bc199cbc0383d0f1e4398beb9d3c668022a4144ddfc2e68"} Nov 04 01:39:34.145281 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:34.145112 7059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acee6a86a515c4346bc199cbc0383d0f1e4398beb9d3c668022a4144ddfc2e68" Nov 04 01:39:36.483974 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:39:36.483904 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:39:40.782327 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:40.782295 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nxfs6"] Nov 04 01:39:40.782802 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:40.782786 7059 cpu_manager.go:414] "containerMap: removing container" podUID="13704caf-55c8-4c4c-985f-da9755654491" containerName="revoke-keys" Nov 04 01:39:40.782855 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:40.782803 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13704caf-55c8-4c4c-985f-da9755654491" containerName="revoke-keys" Nov 04 01:39:40.782933 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:40.782913 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="13704caf-55c8-4c4c-985f-da9755654491" containerName="revoke-keys" Nov 04 01:39:40.786150 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:40.786130 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nxfs6" Nov 04 01:39:40.792123 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:40.792093 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:39:40.792261 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:40.792179 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:39:40.798337 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:40.798307 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nxfs6"] Nov 04 01:39:40.810534 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:40.810453 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6ff9290-277d-4b7e-8498-75d2064c6749-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nxfs6\" (UID: \"e6ff9290-277d-4b7e-8498-75d2064c6749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nxfs6" Nov 04 01:39:40.810534 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:40.810503 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqfps\" (UniqueName: \"kubernetes.io/projected/e6ff9290-277d-4b7e-8498-75d2064c6749-kube-api-access-gqfps\") pod \"maas-api-revoke-sub-3d4156f1-nxfs6\" (UID: \"e6ff9290-277d-4b7e-8498-75d2064c6749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nxfs6" Nov 04 01:39:40.810712 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:40.810652 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6ff9290-277d-4b7e-8498-75d2064c6749-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nxfs6\" (UID: \"e6ff9290-277d-4b7e-8498-75d2064c6749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nxfs6" Nov 04 01:39:40.911353 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:40.911319 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6ff9290-277d-4b7e-8498-75d2064c6749-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nxfs6\" (UID: \"e6ff9290-277d-4b7e-8498-75d2064c6749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nxfs6" Nov 04 01:39:40.911501 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:40.911436 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6ff9290-277d-4b7e-8498-75d2064c6749-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nxfs6\" (UID: \"e6ff9290-277d-4b7e-8498-75d2064c6749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nxfs6" Nov 04 01:39:40.911501 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:40.911465 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqfps\" (UniqueName: \"kubernetes.io/projected/e6ff9290-277d-4b7e-8498-75d2064c6749-kube-api-access-gqfps\") pod \"maas-api-revoke-sub-3d4156f1-nxfs6\" (UID: \"e6ff9290-277d-4b7e-8498-75d2064c6749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nxfs6" Nov 04 01:39:40.912018 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:40.911996 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6ff9290-277d-4b7e-8498-75d2064c6749-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nxfs6\" (UID: \"e6ff9290-277d-4b7e-8498-75d2064c6749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nxfs6" Nov 04 01:39:40.913845 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:40.913822 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6ff9290-277d-4b7e-8498-75d2064c6749-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nxfs6\" (UID: \"e6ff9290-277d-4b7e-8498-75d2064c6749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nxfs6" Nov 04 01:39:40.919910 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:40.919885 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqfps\" (UniqueName: \"kubernetes.io/projected/e6ff9290-277d-4b7e-8498-75d2064c6749-kube-api-access-gqfps\") pod \"maas-api-revoke-sub-3d4156f1-nxfs6\" (UID: \"e6ff9290-277d-4b7e-8498-75d2064c6749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nxfs6" Nov 04 01:39:41.104226 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:41.104151 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nxfs6" Nov 04 01:39:41.380772 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:41.380742 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nxfs6"] Nov 04 01:39:41.381495 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:39:41.381469 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ff9290_277d_4b7e_8498_75d2064c6749.slice/crio-4d206eca041265944e61fff4f1481ae59e6e476ddaa42a091339154366d83947 WatchSource:0}: Error finding container 4d206eca041265944e61fff4f1481ae59e6e476ddaa42a091339154366d83947: Status 404 returned error can't find the container with id 4d206eca041265944e61fff4f1481ae59e6e476ddaa42a091339154366d83947 Nov 04 01:39:42.193550 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:42.193517 7059 generic.go:356] "Generic (PLEG): container finished" podID="e6ff9290-277d-4b7e-8498-75d2064c6749" containerID="c6c0cecac85e2ef57309a6d66f8e5aa9c8bfc426a413843b94dbc6a0b187c18e" exitCode=0 Nov 04 01:39:42.193867 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:42.193574 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nxfs6" event={"ID":"e6ff9290-277d-4b7e-8498-75d2064c6749","Type":"ContainerDied","Data":"c6c0cecac85e2ef57309a6d66f8e5aa9c8bfc426a413843b94dbc6a0b187c18e"} Nov 04 01:39:42.193867 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:42.193623 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nxfs6" event={"ID":"e6ff9290-277d-4b7e-8498-75d2064c6749","Type":"ContainerStarted","Data":"4d206eca041265944e61fff4f1481ae59e6e476ddaa42a091339154366d83947"} Nov 04 01:39:43.338091 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:43.338064 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nxfs6" Nov 04 01:39:43.439181 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:43.439150 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6ff9290-277d-4b7e-8498-75d2064c6749-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6ff9290-277d-4b7e-8498-75d2064c6749-maas-api-openshift-ca\") pod \"e6ff9290-277d-4b7e-8498-75d2064c6749\" (UID: \"e6ff9290-277d-4b7e-8498-75d2064c6749\") " Nov 04 01:39:43.439366 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:43.439261 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6ff9290-277d-4b7e-8498-75d2064c6749-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6ff9290-277d-4b7e-8498-75d2064c6749-maas-api-xks-ca\") pod \"e6ff9290-277d-4b7e-8498-75d2064c6749\" (UID: \"e6ff9290-277d-4b7e-8498-75d2064c6749\") " Nov 04 01:39:43.439366 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:43.439316 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6ff9290-277d-4b7e-8498-75d2064c6749-kube-api-access-gqfps\" (UniqueName: \"kubernetes.io/projected/e6ff9290-277d-4b7e-8498-75d2064c6749-kube-api-access-gqfps\") pod \"e6ff9290-277d-4b7e-8498-75d2064c6749\" (UID: \"e6ff9290-277d-4b7e-8498-75d2064c6749\") " Nov 04 01:39:43.439514 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:43.439488 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6ff9290-277d-4b7e-8498-75d2064c6749-maas-api-openshift-ca" pod "e6ff9290-277d-4b7e-8498-75d2064c6749" (UID: "e6ff9290-277d-4b7e-8498-75d2064c6749"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:39:43.441581 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:43.441555 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6ff9290-277d-4b7e-8498-75d2064c6749-maas-api-xks-ca" pod "e6ff9290-277d-4b7e-8498-75d2064c6749" (UID: "e6ff9290-277d-4b7e-8498-75d2064c6749"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:39:43.441697 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:43.441557 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6ff9290-277d-4b7e-8498-75d2064c6749-kube-api-access-gqfps" pod "e6ff9290-277d-4b7e-8498-75d2064c6749" (UID: "e6ff9290-277d-4b7e-8498-75d2064c6749"). InnerVolumeSpecName "kube-api-access-gqfps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:39:43.540306 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:43.540217 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6ff9290-277d-4b7e-8498-75d2064c6749-maas-api-xks-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:43.540306 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:43.540252 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqfps\" (UniqueName: \"kubernetes.io/projected/e6ff9290-277d-4b7e-8498-75d2064c6749-kube-api-access-gqfps\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:43.540306 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:43.540266 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6ff9290-277d-4b7e-8498-75d2064c6749-maas-api-openshift-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:44.206463 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:44.206435 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nxfs6" Nov 04 01:39:44.206615 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:44.206434 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nxfs6" event={"ID":"e6ff9290-277d-4b7e-8498-75d2064c6749","Type":"ContainerDied","Data":"4d206eca041265944e61fff4f1481ae59e6e476ddaa42a091339154366d83947"} Nov 04 01:39:44.206615 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:44.206560 7059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d206eca041265944e61fff4f1481ae59e6e476ddaa42a091339154366d83947" Nov 04 01:39:59.184209 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:59.184170 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nxfs6"] Nov 04 01:39:59.184729 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:59.184226 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nxfs6"] Nov 04 01:39:59.197510 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:59.197483 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-68x7h"] Nov 04 01:39:59.197918 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:59.197907 7059 cpu_manager.go:414] "containerMap: removing container" podUID="e6ff9290-277d-4b7e-8498-75d2064c6749" containerName="revoke-keys" Nov 04 01:39:59.197989 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:59.197935 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6ff9290-277d-4b7e-8498-75d2064c6749" containerName="revoke-keys" Nov 04 01:39:59.198033 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:59.198024 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6ff9290-277d-4b7e-8498-75d2064c6749" containerName="revoke-keys" Nov 04 01:39:59.201604 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:59.201586 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-68x7h" Nov 04 01:39:59.206578 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:59.206553 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:39:59.207456 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:59.207420 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:39:59.209216 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:59.209190 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-68x7h"] Nov 04 01:39:59.278785 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:59.278754 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4013db06-77aa-4b81-b63a-f78102326042-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-68x7h\" (UID: \"4013db06-77aa-4b81-b63a-f78102326042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-68x7h" Nov 04 01:39:59.278942 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:59.278804 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64969\" (UniqueName: \"kubernetes.io/projected/4013db06-77aa-4b81-b63a-f78102326042-kube-api-access-64969\") pod \"maas-api-revoke-sub-3d4156f1-68x7h\" (UID: \"4013db06-77aa-4b81-b63a-f78102326042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-68x7h" Nov 04 01:39:59.278942 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:59.278876 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4013db06-77aa-4b81-b63a-f78102326042-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-68x7h\" (UID: \"4013db06-77aa-4b81-b63a-f78102326042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-68x7h" Nov 04 01:39:59.380516 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:59.380484 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4013db06-77aa-4b81-b63a-f78102326042-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-68x7h\" (UID: \"4013db06-77aa-4b81-b63a-f78102326042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-68x7h" Nov 04 01:39:59.380671 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:59.380529 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64969\" (UniqueName: \"kubernetes.io/projected/4013db06-77aa-4b81-b63a-f78102326042-kube-api-access-64969\") pod \"maas-api-revoke-sub-3d4156f1-68x7h\" (UID: \"4013db06-77aa-4b81-b63a-f78102326042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-68x7h" Nov 04 01:39:59.380671 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:59.380559 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4013db06-77aa-4b81-b63a-f78102326042-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-68x7h\" (UID: \"4013db06-77aa-4b81-b63a-f78102326042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-68x7h" Nov 04 01:39:59.381268 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:59.381250 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4013db06-77aa-4b81-b63a-f78102326042-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-68x7h\" (UID: \"4013db06-77aa-4b81-b63a-f78102326042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-68x7h" Nov 04 01:39:59.383146 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:59.383129 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4013db06-77aa-4b81-b63a-f78102326042-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-68x7h\" (UID: \"4013db06-77aa-4b81-b63a-f78102326042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-68x7h" Nov 04 01:39:59.390514 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:59.390491 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64969\" (UniqueName: \"kubernetes.io/projected/4013db06-77aa-4b81-b63a-f78102326042-kube-api-access-64969\") pod \"maas-api-revoke-sub-3d4156f1-68x7h\" (UID: \"4013db06-77aa-4b81-b63a-f78102326042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-68x7h" Nov 04 01:39:59.519963 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:59.519887 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-68x7h" Nov 04 01:39:59.799855 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:39:59.799828 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-68x7h"] Nov 04 01:39:59.802026 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:39:59.801994 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4013db06_77aa_4b81_b63a_f78102326042.slice/crio-c97f237c022cd948de505219e817186fe527ade969e6d734280072e45d624cd4 WatchSource:0}: Error finding container c97f237c022cd948de505219e817186fe527ade969e6d734280072e45d624cd4: Status 404 returned error can't find the container with id c97f237c022cd948de505219e817186fe527ade969e6d734280072e45d624cd4 Nov 04 01:40:00.353396 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:00.353307 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-68x7h" event={"ID":"4013db06-77aa-4b81-b63a-f78102326042","Type":"ContainerStarted","Data":"eb94f69400fb883910cdf4fbae9d847260e35012b11d77070bac9bccfe6ff84f"} Nov 04 01:40:00.353396 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:00.353373 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-68x7h" event={"ID":"4013db06-77aa-4b81-b63a-f78102326042","Type":"ContainerStarted","Data":"c97f237c022cd948de505219e817186fe527ade969e6d734280072e45d624cd4"} Nov 04 01:40:00.371156 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:00.371106 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-68x7h" podStartSLOduration=1.371092513 podStartE2EDuration="1.371092513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:40:00.370451038 +0000 UTC m=+2485.312299941" watchObservedRunningTime="2026-11-04 01:40:00.371092513 +0000 UTC m=+2485.312941416" Nov 04 01:40:01.176190 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:01.176153 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6ff9290-277d-4b7e-8498-75d2064c6749" path="/var/lib/kubelet/pods/e6ff9290-277d-4b7e-8498-75d2064c6749/volumes" Nov 04 01:40:01.360472 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:01.360436 7059 generic.go:356] "Generic (PLEG): container finished" podID="4013db06-77aa-4b81-b63a-f78102326042" containerID="eb94f69400fb883910cdf4fbae9d847260e35012b11d77070bac9bccfe6ff84f" exitCode=0 Nov 04 01:40:01.360863 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:01.360523 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-68x7h" event={"ID":"4013db06-77aa-4b81-b63a-f78102326042","Type":"ContainerDied","Data":"eb94f69400fb883910cdf4fbae9d847260e35012b11d77070bac9bccfe6ff84f"} Nov 04 01:40:02.515569 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:02.515545 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-68x7h" Nov 04 01:40:02.610801 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:02.610770 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4013db06-77aa-4b81-b63a-f78102326042-kube-api-access-64969\" (UniqueName: \"kubernetes.io/projected/4013db06-77aa-4b81-b63a-f78102326042-kube-api-access-64969\") pod \"4013db06-77aa-4b81-b63a-f78102326042\" (UID: \"4013db06-77aa-4b81-b63a-f78102326042\") " Nov 04 01:40:02.611043 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:02.610904 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4013db06-77aa-4b81-b63a-f78102326042-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4013db06-77aa-4b81-b63a-f78102326042-maas-api-openshift-ca\") pod \"4013db06-77aa-4b81-b63a-f78102326042\" (UID: \"4013db06-77aa-4b81-b63a-f78102326042\") " Nov 04 01:40:02.611043 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:02.611004 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4013db06-77aa-4b81-b63a-f78102326042-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4013db06-77aa-4b81-b63a-f78102326042-maas-api-xks-ca\") pod \"4013db06-77aa-4b81-b63a-f78102326042\" (UID: \"4013db06-77aa-4b81-b63a-f78102326042\") " Nov 04 01:40:02.611480 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:02.611260 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4013db06-77aa-4b81-b63a-f78102326042-maas-api-openshift-ca" pod "4013db06-77aa-4b81-b63a-f78102326042" (UID: "4013db06-77aa-4b81-b63a-f78102326042"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:40:02.611480 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:02.611429 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4013db06-77aa-4b81-b63a-f78102326042-maas-api-openshift-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:02.614025 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:02.613807 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4013db06-77aa-4b81-b63a-f78102326042-maas-api-xks-ca" pod "4013db06-77aa-4b81-b63a-f78102326042" (UID: "4013db06-77aa-4b81-b63a-f78102326042"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:40:02.615295 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:02.615271 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4013db06-77aa-4b81-b63a-f78102326042-kube-api-access-64969" pod "4013db06-77aa-4b81-b63a-f78102326042" (UID: "4013db06-77aa-4b81-b63a-f78102326042"). InnerVolumeSpecName "kube-api-access-64969". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:40:02.712642 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:02.712615 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4013db06-77aa-4b81-b63a-f78102326042-maas-api-xks-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:02.712642 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:02.712639 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64969\" (UniqueName: \"kubernetes.io/projected/4013db06-77aa-4b81-b63a-f78102326042-kube-api-access-64969\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:03.373174 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:03.373138 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-68x7h" event={"ID":"4013db06-77aa-4b81-b63a-f78102326042","Type":"ContainerDied","Data":"c97f237c022cd948de505219e817186fe527ade969e6d734280072e45d624cd4"} Nov 04 01:40:03.373174 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:03.373160 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-68x7h" Nov 04 01:40:03.373425 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:03.373185 7059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c97f237c022cd948de505219e817186fe527ade969e6d734280072e45d624cd4" Nov 04 01:40:06.479509 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:40:06.479470 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:40:22.237759 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.237727 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n9tzj"] Nov 04 01:40:22.240183 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.238372 7059 cpu_manager.go:414] "containerMap: removing container" podUID="4013db06-77aa-4b81-b63a-f78102326042" containerName="revoke-keys" Nov 04 01:40:22.240183 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.238390 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4013db06-77aa-4b81-b63a-f78102326042" containerName="revoke-keys" Nov 04 01:40:22.240183 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.238492 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="4013db06-77aa-4b81-b63a-f78102326042" containerName="revoke-keys" Nov 04 01:40:22.241479 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.241463 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n9tzj" Nov 04 01:40:22.246876 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.246697 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:40:22.246876 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.246758 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:40:22.249868 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.249498 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n9tzj"] Nov 04 01:40:22.336800 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.336764 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np47s\" (UniqueName: \"kubernetes.io/projected/a54b864e-1ded-486d-b8b1-38d87c61dd53-kube-api-access-np47s\") pod \"maas-api-revoke-sub-e2fc9c5b-n9tzj\" (UID: \"a54b864e-1ded-486d-b8b1-38d87c61dd53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n9tzj" Nov 04 01:40:22.336983 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.336806 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a54b864e-1ded-486d-b8b1-38d87c61dd53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-n9tzj\" (UID: \"a54b864e-1ded-486d-b8b1-38d87c61dd53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n9tzj" Nov 04 01:40:22.336983 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.336833 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a54b864e-1ded-486d-b8b1-38d87c61dd53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-n9tzj\" (UID: \"a54b864e-1ded-486d-b8b1-38d87c61dd53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n9tzj" Nov 04 01:40:22.437779 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.437748 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np47s\" (UniqueName: \"kubernetes.io/projected/a54b864e-1ded-486d-b8b1-38d87c61dd53-kube-api-access-np47s\") pod \"maas-api-revoke-sub-e2fc9c5b-n9tzj\" (UID: \"a54b864e-1ded-486d-b8b1-38d87c61dd53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n9tzj" Nov 04 01:40:22.437915 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.437786 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a54b864e-1ded-486d-b8b1-38d87c61dd53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-n9tzj\" (UID: \"a54b864e-1ded-486d-b8b1-38d87c61dd53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n9tzj" Nov 04 01:40:22.437915 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.437807 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a54b864e-1ded-486d-b8b1-38d87c61dd53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-n9tzj\" (UID: \"a54b864e-1ded-486d-b8b1-38d87c61dd53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n9tzj" Nov 04 01:40:22.438447 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.438429 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a54b864e-1ded-486d-b8b1-38d87c61dd53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-n9tzj\" (UID: \"a54b864e-1ded-486d-b8b1-38d87c61dd53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n9tzj" Nov 04 01:40:22.440298 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.440275 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a54b864e-1ded-486d-b8b1-38d87c61dd53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-n9tzj\" (UID: \"a54b864e-1ded-486d-b8b1-38d87c61dd53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n9tzj" Nov 04 01:40:22.446075 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.446054 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np47s\" (UniqueName: \"kubernetes.io/projected/a54b864e-1ded-486d-b8b1-38d87c61dd53-kube-api-access-np47s\") pod \"maas-api-revoke-sub-e2fc9c5b-n9tzj\" (UID: \"a54b864e-1ded-486d-b8b1-38d87c61dd53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n9tzj" Nov 04 01:40:22.557528 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.557466 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n9tzj" Nov 04 01:40:22.662811 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.662759 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct"] Nov 04 01:40:22.663388 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.663328 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct" podUID="1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd" containerName="discovery" containerID="cri-o://53099f172a42ae74621e1c7a36702e25cf99cae32b19eb3a0d4c502a62bed227" gracePeriod=30 Nov 04 01:40:22.835955 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.835898 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n9tzj"] Nov 04 01:40:22.837634 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:40:22.837608 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda54b864e_1ded_486d_b8b1_38d87c61dd53.slice/crio-9a5b00e479a5dd52d749ee62dfce0f93f6a8e5cc9a524451d6bcb0d3eb34507d WatchSource:0}: Error finding container 9a5b00e479a5dd52d749ee62dfce0f93f6a8e5cc9a524451d6bcb0d3eb34507d: Status 404 returned error can't find the container with id 9a5b00e479a5dd52d749ee62dfce0f93f6a8e5cc9a524451d6bcb0d3eb34507d Nov 04 01:40:22.908107 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.908085 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct" Nov 04 01:40:22.943319 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.943294 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-kube-api-access-gjd42\" (UniqueName: \"kubernetes.io/projected/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-kube-api-access-gjd42\") pod \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\" (UID: \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\") " Nov 04 01:40:22.943458 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.943332 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-cacerts\" (UniqueName: \"kubernetes.io/secret/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-cacerts\") pod \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\" (UID: \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\") " Nov 04 01:40:22.943458 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.943367 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-istio-csr-dns-cert\") pod \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\" (UID: \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\") " Nov 04 01:40:22.943458 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.943438 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-istio-csr-ca-configmap\") pod \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\" (UID: \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\") " Nov 04 01:40:22.943638 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.943465 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-istio-kubeconfig\") pod \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\" (UID: \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\") " Nov 04 01:40:22.943638 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.943539 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-istio-token\" (UniqueName: \"kubernetes.io/projected/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-istio-token\") pod \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\" (UID: \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\") " Nov 04 01:40:22.943638 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.943605 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-local-certs\") pod \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\" (UID: \"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd\") " Nov 04 01:40:22.944482 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.944438 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-istio-csr-ca-configmap" pod "1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd" (UID: "1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:40:22.946319 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.946288 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-istio-csr-dns-cert" pod "1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd" (UID: "1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:40:22.946416 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.946334 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-local-certs" pod "1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd" (UID: "1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:40:22.946416 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.946373 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-cacerts" pod "1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd" (UID: "1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:40:22.946912 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.946633 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-istio-token" pod "1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd" (UID: "1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:40:22.946912 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.946748 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-kube-api-access-gjd42" pod "1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd" (UID: "1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd"). InnerVolumeSpecName "kube-api-access-gjd42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:40:22.947055 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:22.946991 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-istio-kubeconfig" pod "1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd" (UID: "1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:40:23.045414 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:23.045390 7059 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-cacerts\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:23.045414 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:23.045413 7059 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-istio-csr-dns-cert\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:23.045553 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:23.045424 7059 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-istio-csr-ca-configmap\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:23.045553 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:23.045436 7059 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-istio-kubeconfig\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:23.045553 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:23.045445 7059 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-istio-token\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:23.045553 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:23.045453 7059 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-local-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:23.045553 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:23.045462 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjd42\" (UniqueName: \"kubernetes.io/projected/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd-kube-api-access-gjd42\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:23.536992 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:23.536958 7059 generic.go:356] "Generic (PLEG): container finished" podID="1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd" containerID="53099f172a42ae74621e1c7a36702e25cf99cae32b19eb3a0d4c502a62bed227" exitCode=0 Nov 04 01:40:23.537392 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:23.537005 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct" Nov 04 01:40:23.537392 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:23.537046 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct" event={"ID":"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd","Type":"ContainerDied","Data":"53099f172a42ae74621e1c7a36702e25cf99cae32b19eb3a0d4c502a62bed227"} Nov 04 01:40:23.537392 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:23.537105 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct" event={"ID":"1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd","Type":"ContainerDied","Data":"6ac55464ab0e26f7b19d5473faefe83670eeff8b5008db524d363630f64e07ca"} Nov 04 01:40:23.537392 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:23.537130 7059 scope.go:122] "RemoveContainer" containerID="53099f172a42ae74621e1c7a36702e25cf99cae32b19eb3a0d4c502a62bed227" Nov 04 01:40:23.538960 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:23.538902 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n9tzj" event={"ID":"a54b864e-1ded-486d-b8b1-38d87c61dd53","Type":"ContainerStarted","Data":"ef5c88ff7b204a76eca53d37a74c8aed2888c81570a3f62cd23b3586e8ba05ce"} Nov 04 01:40:23.539083 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:23.538988 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n9tzj" event={"ID":"a54b864e-1ded-486d-b8b1-38d87c61dd53","Type":"ContainerStarted","Data":"9a5b00e479a5dd52d749ee62dfce0f93f6a8e5cc9a524451d6bcb0d3eb34507d"} Nov 04 01:40:23.547971 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:23.547947 7059 scope.go:122] "RemoveContainer" containerID="53099f172a42ae74621e1c7a36702e25cf99cae32b19eb3a0d4c502a62bed227" Nov 04 01:40:23.548258 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:40:23.548237 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53099f172a42ae74621e1c7a36702e25cf99cae32b19eb3a0d4c502a62bed227\": container with ID starting with 53099f172a42ae74621e1c7a36702e25cf99cae32b19eb3a0d4c502a62bed227 not found: ID does not exist" containerID="53099f172a42ae74621e1c7a36702e25cf99cae32b19eb3a0d4c502a62bed227" Nov 04 01:40:23.548332 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:23.548269 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53099f172a42ae74621e1c7a36702e25cf99cae32b19eb3a0d4c502a62bed227"} err="failed to get container status \"53099f172a42ae74621e1c7a36702e25cf99cae32b19eb3a0d4c502a62bed227\": rpc error: code = NotFound desc = could not find container \"53099f172a42ae74621e1c7a36702e25cf99cae32b19eb3a0d4c502a62bed227\": container with ID starting with 53099f172a42ae74621e1c7a36702e25cf99cae32b19eb3a0d4c502a62bed227 not found: ID does not exist" Nov 04 01:40:23.559330 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:23.559286 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n9tzj" podStartSLOduration=1.5592755249999999 podStartE2EDuration="1.559275525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:40:23.556665415 +0000 UTC m=+2508.498514320" watchObservedRunningTime="2026-11-04 01:40:23.559275525 +0000 UTC m=+2508.501124429" Nov 04 01:40:23.571413 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:23.571388 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct"] Nov 04 01:40:23.574981 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:23.574961 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4snct"] Nov 04 01:40:24.546982 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:24.546944 7059 generic.go:356] "Generic (PLEG): container finished" podID="a54b864e-1ded-486d-b8b1-38d87c61dd53" containerID="ef5c88ff7b204a76eca53d37a74c8aed2888c81570a3f62cd23b3586e8ba05ce" exitCode=0 Nov 04 01:40:24.547442 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:24.547029 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n9tzj" event={"ID":"a54b864e-1ded-486d-b8b1-38d87c61dd53","Type":"ContainerDied","Data":"ef5c88ff7b204a76eca53d37a74c8aed2888c81570a3f62cd23b3586e8ba05ce"} Nov 04 01:40:25.176641 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:25.176607 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd" path="/var/lib/kubelet/pods/1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd/volumes" Nov 04 01:40:25.687402 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:25.687377 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n9tzj" Nov 04 01:40:25.771747 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:25.771718 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a54b864e-1ded-486d-b8b1-38d87c61dd53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a54b864e-1ded-486d-b8b1-38d87c61dd53-maas-api-openshift-ca\") pod \"a54b864e-1ded-486d-b8b1-38d87c61dd53\" (UID: \"a54b864e-1ded-486d-b8b1-38d87c61dd53\") " Nov 04 01:40:25.771897 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:25.771779 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a54b864e-1ded-486d-b8b1-38d87c61dd53-kube-api-access-np47s\" (UniqueName: \"kubernetes.io/projected/a54b864e-1ded-486d-b8b1-38d87c61dd53-kube-api-access-np47s\") pod \"a54b864e-1ded-486d-b8b1-38d87c61dd53\" (UID: \"a54b864e-1ded-486d-b8b1-38d87c61dd53\") " Nov 04 01:40:25.771897 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:25.771807 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a54b864e-1ded-486d-b8b1-38d87c61dd53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a54b864e-1ded-486d-b8b1-38d87c61dd53-maas-api-xks-ca\") pod \"a54b864e-1ded-486d-b8b1-38d87c61dd53\" (UID: \"a54b864e-1ded-486d-b8b1-38d87c61dd53\") " Nov 04 01:40:25.772120 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:25.772091 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a54b864e-1ded-486d-b8b1-38d87c61dd53-maas-api-openshift-ca" pod "a54b864e-1ded-486d-b8b1-38d87c61dd53" (UID: "a54b864e-1ded-486d-b8b1-38d87c61dd53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:40:25.773998 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:25.773976 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a54b864e-1ded-486d-b8b1-38d87c61dd53-maas-api-xks-ca" pod "a54b864e-1ded-486d-b8b1-38d87c61dd53" (UID: "a54b864e-1ded-486d-b8b1-38d87c61dd53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:40:25.774117 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:25.774093 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a54b864e-1ded-486d-b8b1-38d87c61dd53-kube-api-access-np47s" pod "a54b864e-1ded-486d-b8b1-38d87c61dd53" (UID: "a54b864e-1ded-486d-b8b1-38d87c61dd53"). InnerVolumeSpecName "kube-api-access-np47s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:40:25.872826 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:25.872770 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a54b864e-1ded-486d-b8b1-38d87c61dd53-maas-api-openshift-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:25.872826 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:25.872794 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np47s\" (UniqueName: \"kubernetes.io/projected/a54b864e-1ded-486d-b8b1-38d87c61dd53-kube-api-access-np47s\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:25.872826 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:25.872804 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a54b864e-1ded-486d-b8b1-38d87c61dd53-maas-api-xks-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:26.560956 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:26.560909 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n9tzj" Nov 04 01:40:26.561122 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:26.560952 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n9tzj" event={"ID":"a54b864e-1ded-486d-b8b1-38d87c61dd53","Type":"ContainerDied","Data":"9a5b00e479a5dd52d749ee62dfce0f93f6a8e5cc9a524451d6bcb0d3eb34507d"} Nov 04 01:40:26.561122 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:26.560998 7059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a5b00e479a5dd52d749ee62dfce0f93f6a8e5cc9a524451d6bcb0d3eb34507d" Nov 04 01:40:32.439546 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:32.439504 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xsc4l"] Nov 04 01:40:32.440196 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:32.440175 7059 cpu_manager.go:414] "containerMap: removing container" podUID="a54b864e-1ded-486d-b8b1-38d87c61dd53" containerName="revoke-keys" Nov 04 01:40:32.440280 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:32.440198 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a54b864e-1ded-486d-b8b1-38d87c61dd53" containerName="revoke-keys" Nov 04 01:40:32.440280 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:32.440256 7059 cpu_manager.go:414] "containerMap: removing container" podUID="1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd" containerName="discovery" Nov 04 01:40:32.440280 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:32.440264 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd" containerName="discovery" Nov 04 01:40:32.440439 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:32.440384 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ab0633d-d9de-42ab-a0f6-d64bfe8ecafd" containerName="discovery" Nov 04 01:40:32.440439 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:32.440402 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="a54b864e-1ded-486d-b8b1-38d87c61dd53" containerName="revoke-keys" Nov 04 01:40:32.443492 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:32.443470 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xsc4l" Nov 04 01:40:32.448032 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:32.448000 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:40:32.448158 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:32.448123 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:40:32.451287 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:32.451251 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xsc4l"] Nov 04 01:40:32.535117 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:32.535082 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8dzx\" (UniqueName: \"kubernetes.io/projected/30e20314-9fdb-47a1-bb32-516965d4e310-kube-api-access-z8dzx\") pod \"maas-api-revoke-sub-5a29efb3-xsc4l\" (UID: \"30e20314-9fdb-47a1-bb32-516965d4e310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xsc4l" Nov 04 01:40:32.535254 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:32.535211 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30e20314-9fdb-47a1-bb32-516965d4e310-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-xsc4l\" (UID: \"30e20314-9fdb-47a1-bb32-516965d4e310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xsc4l" Nov 04 01:40:32.535315 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:32.535264 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30e20314-9fdb-47a1-bb32-516965d4e310-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-xsc4l\" (UID: \"30e20314-9fdb-47a1-bb32-516965d4e310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xsc4l" Nov 04 01:40:32.636064 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:32.636022 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8dzx\" (UniqueName: \"kubernetes.io/projected/30e20314-9fdb-47a1-bb32-516965d4e310-kube-api-access-z8dzx\") pod \"maas-api-revoke-sub-5a29efb3-xsc4l\" (UID: \"30e20314-9fdb-47a1-bb32-516965d4e310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xsc4l" Nov 04 01:40:32.636244 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:32.636149 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30e20314-9fdb-47a1-bb32-516965d4e310-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-xsc4l\" (UID: \"30e20314-9fdb-47a1-bb32-516965d4e310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xsc4l" Nov 04 01:40:32.636244 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:32.636215 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30e20314-9fdb-47a1-bb32-516965d4e310-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-xsc4l\" (UID: \"30e20314-9fdb-47a1-bb32-516965d4e310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xsc4l" Nov 04 01:40:32.636914 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:32.636888 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30e20314-9fdb-47a1-bb32-516965d4e310-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-xsc4l\" (UID: \"30e20314-9fdb-47a1-bb32-516965d4e310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xsc4l" Nov 04 01:40:32.638699 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:32.638678 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30e20314-9fdb-47a1-bb32-516965d4e310-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-xsc4l\" (UID: \"30e20314-9fdb-47a1-bb32-516965d4e310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xsc4l" Nov 04 01:40:32.643972 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:32.643953 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8dzx\" (UniqueName: \"kubernetes.io/projected/30e20314-9fdb-47a1-bb32-516965d4e310-kube-api-access-z8dzx\") pod \"maas-api-revoke-sub-5a29efb3-xsc4l\" (UID: \"30e20314-9fdb-47a1-bb32-516965d4e310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xsc4l" Nov 04 01:40:32.761073 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:32.761014 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xsc4l" Nov 04 01:40:33.038807 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:33.038777 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xsc4l"] Nov 04 01:40:33.039915 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:40:33.039886 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e20314_9fdb_47a1_bb32_516965d4e310.slice/crio-f79d67b429f5dec5830b18b720da61322c427a52628f95700b7e5fca772015a8 WatchSource:0}: Error finding container f79d67b429f5dec5830b18b720da61322c427a52628f95700b7e5fca772015a8: Status 404 returned error can't find the container with id f79d67b429f5dec5830b18b720da61322c427a52628f95700b7e5fca772015a8 Nov 04 01:40:33.605640 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:33.605529 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xsc4l" event={"ID":"30e20314-9fdb-47a1-bb32-516965d4e310","Type":"ContainerStarted","Data":"220ffb6d915e850f755f09745cc8fe08cf1d393b287def7740c8a47db008d99d"} Nov 04 01:40:33.605640 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:33.605605 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xsc4l" event={"ID":"30e20314-9fdb-47a1-bb32-516965d4e310","Type":"ContainerStarted","Data":"f79d67b429f5dec5830b18b720da61322c427a52628f95700b7e5fca772015a8"} Nov 04 01:40:33.625668 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:33.625610 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xsc4l" podStartSLOduration=1.625593426 podStartE2EDuration="1.625593426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:40:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:40:33.624168374 +0000 UTC m=+2518.566017273" watchObservedRunningTime="2026-11-04 01:40:33.625593426 +0000 UTC m=+2518.567442326" Nov 04 01:40:34.614217 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:34.614181 7059 generic.go:356] "Generic (PLEG): container finished" podID="30e20314-9fdb-47a1-bb32-516965d4e310" containerID="220ffb6d915e850f755f09745cc8fe08cf1d393b287def7740c8a47db008d99d" exitCode=0 Nov 04 01:40:34.614710 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:34.614403 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xsc4l" event={"ID":"30e20314-9fdb-47a1-bb32-516965d4e310","Type":"ContainerDied","Data":"220ffb6d915e850f755f09745cc8fe08cf1d393b287def7740c8a47db008d99d"} Nov 04 01:40:36.016259 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:36.016222 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xsc4l" Nov 04 01:40:36.033203 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:36.032877 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30e20314-9fdb-47a1-bb32-516965d4e310-kube-api-access-z8dzx\" (UniqueName: \"kubernetes.io/projected/30e20314-9fdb-47a1-bb32-516965d4e310-kube-api-access-z8dzx\") pod \"30e20314-9fdb-47a1-bb32-516965d4e310\" (UID: \"30e20314-9fdb-47a1-bb32-516965d4e310\") " Nov 04 01:40:36.033203 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:36.032954 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30e20314-9fdb-47a1-bb32-516965d4e310-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30e20314-9fdb-47a1-bb32-516965d4e310-maas-api-openshift-ca\") pod \"30e20314-9fdb-47a1-bb32-516965d4e310\" (UID: \"30e20314-9fdb-47a1-bb32-516965d4e310\") " Nov 04 01:40:36.033203 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:36.033029 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30e20314-9fdb-47a1-bb32-516965d4e310-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30e20314-9fdb-47a1-bb32-516965d4e310-maas-api-xks-ca\") pod \"30e20314-9fdb-47a1-bb32-516965d4e310\" (UID: \"30e20314-9fdb-47a1-bb32-516965d4e310\") " Nov 04 01:40:36.035464 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:36.035434 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30e20314-9fdb-47a1-bb32-516965d4e310-maas-api-openshift-ca" pod "30e20314-9fdb-47a1-bb32-516965d4e310" (UID: "30e20314-9fdb-47a1-bb32-516965d4e310"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:40:36.055948 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:36.050043 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30e20314-9fdb-47a1-bb32-516965d4e310-kube-api-access-z8dzx" pod "30e20314-9fdb-47a1-bb32-516965d4e310" (UID: "30e20314-9fdb-47a1-bb32-516965d4e310"). InnerVolumeSpecName "kube-api-access-z8dzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:40:36.079946 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:36.073070 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30e20314-9fdb-47a1-bb32-516965d4e310-maas-api-xks-ca" pod "30e20314-9fdb-47a1-bb32-516965d4e310" (UID: "30e20314-9fdb-47a1-bb32-516965d4e310"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:40:36.134569 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:36.134539 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30e20314-9fdb-47a1-bb32-516965d4e310-maas-api-xks-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:36.134569 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:36.134564 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8dzx\" (UniqueName: \"kubernetes.io/projected/30e20314-9fdb-47a1-bb32-516965d4e310-kube-api-access-z8dzx\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:36.134739 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:36.134577 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30e20314-9fdb-47a1-bb32-516965d4e310-maas-api-openshift-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:36.479810 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:40:36.479778 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:40:36.630483 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:36.630453 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xsc4l" Nov 04 01:40:36.630667 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:36.630481 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xsc4l" event={"ID":"30e20314-9fdb-47a1-bb32-516965d4e310","Type":"ContainerDied","Data":"f79d67b429f5dec5830b18b720da61322c427a52628f95700b7e5fca772015a8"} Nov 04 01:40:36.630667 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:36.630531 7059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f79d67b429f5dec5830b18b720da61322c427a52628f95700b7e5fca772015a8" Nov 04 01:40:48.105110 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:48.105019 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-68x7h"] Nov 04 01:40:48.109226 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:48.109194 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-68x7h"] Nov 04 01:40:48.125014 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:48.124978 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vj66h"] Nov 04 01:40:48.125437 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:48.125420 7059 cpu_manager.go:414] "containerMap: removing container" podUID="30e20314-9fdb-47a1-bb32-516965d4e310" containerName="revoke-keys" Nov 04 01:40:48.125437 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:48.125434 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30e20314-9fdb-47a1-bb32-516965d4e310" containerName="revoke-keys" Nov 04 01:40:48.125629 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:48.125530 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="30e20314-9fdb-47a1-bb32-516965d4e310" containerName="revoke-keys" Nov 04 01:40:48.130114 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:48.130091 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vj66h" Nov 04 01:40:48.134163 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:48.134137 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:40:48.134783 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:48.134763 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:40:48.136566 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:48.136540 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vj66h"] Nov 04 01:40:48.248488 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:48.248455 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1b9871-4d63-46d1-ba30-859d7b911d14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vj66h\" (UID: \"1f1b9871-4d63-46d1-ba30-859d7b911d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vj66h" Nov 04 01:40:48.248679 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:48.248521 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1b9871-4d63-46d1-ba30-859d7b911d14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vj66h\" (UID: \"1f1b9871-4d63-46d1-ba30-859d7b911d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vj66h" Nov 04 01:40:48.248679 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:48.248616 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm9wv\" (UniqueName: \"kubernetes.io/projected/1f1b9871-4d63-46d1-ba30-859d7b911d14-kube-api-access-nm9wv\") pod \"maas-api-revoke-sub-3d4156f1-vj66h\" (UID: \"1f1b9871-4d63-46d1-ba30-859d7b911d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vj66h" Nov 04 01:40:48.349640 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:48.349605 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1b9871-4d63-46d1-ba30-859d7b911d14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vj66h\" (UID: \"1f1b9871-4d63-46d1-ba30-859d7b911d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vj66h" Nov 04 01:40:48.349807 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:48.349661 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm9wv\" (UniqueName: \"kubernetes.io/projected/1f1b9871-4d63-46d1-ba30-859d7b911d14-kube-api-access-nm9wv\") pod \"maas-api-revoke-sub-3d4156f1-vj66h\" (UID: \"1f1b9871-4d63-46d1-ba30-859d7b911d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vj66h" Nov 04 01:40:48.349807 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:48.349789 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1b9871-4d63-46d1-ba30-859d7b911d14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vj66h\" (UID: \"1f1b9871-4d63-46d1-ba30-859d7b911d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vj66h" Nov 04 01:40:48.350577 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:48.350543 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1b9871-4d63-46d1-ba30-859d7b911d14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vj66h\" (UID: \"1f1b9871-4d63-46d1-ba30-859d7b911d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vj66h" Nov 04 01:40:48.352192 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:48.352166 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1b9871-4d63-46d1-ba30-859d7b911d14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vj66h\" (UID: \"1f1b9871-4d63-46d1-ba30-859d7b911d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vj66h" Nov 04 01:40:48.358014 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:48.357963 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm9wv\" (UniqueName: \"kubernetes.io/projected/1f1b9871-4d63-46d1-ba30-859d7b911d14-kube-api-access-nm9wv\") pod \"maas-api-revoke-sub-3d4156f1-vj66h\" (UID: \"1f1b9871-4d63-46d1-ba30-859d7b911d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vj66h" Nov 04 01:40:48.457255 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:48.457236 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vj66h" Nov 04 01:40:48.730881 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:48.730851 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vj66h"] Nov 04 01:40:48.732157 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:40:48.732128 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1b9871_4d63_46d1_ba30_859d7b911d14.slice/crio-b08dea0486deee106c66e6d2cac9e02bfabd92f3c3d500d35983365b12d067dd WatchSource:0}: Error finding container b08dea0486deee106c66e6d2cac9e02bfabd92f3c3d500d35983365b12d067dd: Status 404 returned error can't find the container with id b08dea0486deee106c66e6d2cac9e02bfabd92f3c3d500d35983365b12d067dd Nov 04 01:40:49.176374 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:49.176343 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4013db06-77aa-4b81-b63a-f78102326042" path="/var/lib/kubelet/pods/4013db06-77aa-4b81-b63a-f78102326042/volumes" Nov 04 01:40:49.707652 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:49.707622 7059 generic.go:356] "Generic (PLEG): container finished" podID="1f1b9871-4d63-46d1-ba30-859d7b911d14" containerID="f468a813ddb6f363f9e51fe9f05759ee78f491f0e470bbfe5dac7676ff00a6a5" exitCode=0 Nov 04 01:40:49.707797 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:49.707712 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vj66h" event={"ID":"1f1b9871-4d63-46d1-ba30-859d7b911d14","Type":"ContainerDied","Data":"f468a813ddb6f363f9e51fe9f05759ee78f491f0e470bbfe5dac7676ff00a6a5"} Nov 04 01:40:49.707797 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:49.707767 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vj66h" event={"ID":"1f1b9871-4d63-46d1-ba30-859d7b911d14","Type":"ContainerStarted","Data":"b08dea0486deee106c66e6d2cac9e02bfabd92f3c3d500d35983365b12d067dd"} Nov 04 01:40:50.856272 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:50.856245 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vj66h" Nov 04 01:40:50.976223 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:50.976161 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f1b9871-4d63-46d1-ba30-859d7b911d14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1b9871-4d63-46d1-ba30-859d7b911d14-maas-api-openshift-ca\") pod \"1f1b9871-4d63-46d1-ba30-859d7b911d14\" (UID: \"1f1b9871-4d63-46d1-ba30-859d7b911d14\") " Nov 04 01:40:50.976223 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:50.976203 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f1b9871-4d63-46d1-ba30-859d7b911d14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1b9871-4d63-46d1-ba30-859d7b911d14-maas-api-xks-ca\") pod \"1f1b9871-4d63-46d1-ba30-859d7b911d14\" (UID: \"1f1b9871-4d63-46d1-ba30-859d7b911d14\") " Nov 04 01:40:50.976387 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:50.976273 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f1b9871-4d63-46d1-ba30-859d7b911d14-kube-api-access-nm9wv\" (UniqueName: \"kubernetes.io/projected/1f1b9871-4d63-46d1-ba30-859d7b911d14-kube-api-access-nm9wv\") pod \"1f1b9871-4d63-46d1-ba30-859d7b911d14\" (UID: \"1f1b9871-4d63-46d1-ba30-859d7b911d14\") " Nov 04 01:40:50.976551 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:50.976530 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f1b9871-4d63-46d1-ba30-859d7b911d14-maas-api-openshift-ca" pod "1f1b9871-4d63-46d1-ba30-859d7b911d14" (UID: "1f1b9871-4d63-46d1-ba30-859d7b911d14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:40:50.976689 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:50.976666 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1b9871-4d63-46d1-ba30-859d7b911d14-maas-api-openshift-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:50.978474 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:50.978453 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f1b9871-4d63-46d1-ba30-859d7b911d14-maas-api-xks-ca" pod "1f1b9871-4d63-46d1-ba30-859d7b911d14" (UID: "1f1b9871-4d63-46d1-ba30-859d7b911d14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:40:50.978582 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:50.978529 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1b9871-4d63-46d1-ba30-859d7b911d14-kube-api-access-nm9wv" pod "1f1b9871-4d63-46d1-ba30-859d7b911d14" (UID: "1f1b9871-4d63-46d1-ba30-859d7b911d14"). InnerVolumeSpecName "kube-api-access-nm9wv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:40:51.078008 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:51.077982 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1b9871-4d63-46d1-ba30-859d7b911d14-maas-api-xks-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:51.078008 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:51.078007 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm9wv\" (UniqueName: \"kubernetes.io/projected/1f1b9871-4d63-46d1-ba30-859d7b911d14-kube-api-access-nm9wv\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:51.721063 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:51.721036 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vj66h" Nov 04 01:40:51.721258 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:51.721035 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vj66h" event={"ID":"1f1b9871-4d63-46d1-ba30-859d7b911d14","Type":"ContainerDied","Data":"b08dea0486deee106c66e6d2cac9e02bfabd92f3c3d500d35983365b12d067dd"} Nov 04 01:40:51.721258 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:40:51.721156 7059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b08dea0486deee106c66e6d2cac9e02bfabd92f3c3d500d35983365b12d067dd" Nov 04 01:41:06.479545 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:41:06.479500 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:41:08.334418 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:08.334385 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w"] Nov 04 01:41:08.334874 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:08.334860 7059 cpu_manager.go:414] "containerMap: removing container" podUID="1f1b9871-4d63-46d1-ba30-859d7b911d14" containerName="revoke-keys" Nov 04 01:41:08.334938 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:08.334875 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f1b9871-4d63-46d1-ba30-859d7b911d14" containerName="revoke-keys" Nov 04 01:41:08.334991 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:08.334981 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f1b9871-4d63-46d1-ba30-859d7b911d14" containerName="revoke-keys" Nov 04 01:41:08.338508 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:08.338487 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w" Nov 04 01:41:08.353560 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:08.351504 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 04 01:41:08.353699 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:08.352006 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-b9b238-serving-cert\"" Nov 04 01:41:08.353699 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:08.352601 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nb79d\"" Nov 04 01:41:08.357239 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:08.357214 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w"] Nov 04 01:41:08.390610 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:08.390564 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fdbeda1-eb2d-400f-a2d9-82b28334ca60-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w\" (UID: \"7fdbeda1-eb2d-400f-a2d9-82b28334ca60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w" Nov 04 01:41:08.390815 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:08.390626 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fdbeda1-eb2d-400f-a2d9-82b28334ca60-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w\" (UID: \"7fdbeda1-eb2d-400f-a2d9-82b28334ca60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w" Nov 04 01:41:08.390815 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:08.390749 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dcx9\" (UniqueName: \"kubernetes.io/projected/7fdbeda1-eb2d-400f-a2d9-82b28334ca60-kube-api-access-4dcx9\") pod \"maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w\" (UID: \"7fdbeda1-eb2d-400f-a2d9-82b28334ca60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w" Nov 04 01:41:08.390815 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:08.390804 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fdbeda1-eb2d-400f-a2d9-82b28334ca60-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w\" (UID: \"7fdbeda1-eb2d-400f-a2d9-82b28334ca60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w" Nov 04 01:41:08.491872 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:08.491840 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fdbeda1-eb2d-400f-a2d9-82b28334ca60-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w\" (UID: \"7fdbeda1-eb2d-400f-a2d9-82b28334ca60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w" Nov 04 01:41:08.492074 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:08.491882 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fdbeda1-eb2d-400f-a2d9-82b28334ca60-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w\" (UID: \"7fdbeda1-eb2d-400f-a2d9-82b28334ca60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w" Nov 04 01:41:08.492074 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:08.491918 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dcx9\" (UniqueName: \"kubernetes.io/projected/7fdbeda1-eb2d-400f-a2d9-82b28334ca60-kube-api-access-4dcx9\") pod \"maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w\" (UID: \"7fdbeda1-eb2d-400f-a2d9-82b28334ca60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w" Nov 04 01:41:08.492074 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:08.491998 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fdbeda1-eb2d-400f-a2d9-82b28334ca60-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w\" (UID: \"7fdbeda1-eb2d-400f-a2d9-82b28334ca60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w" Nov 04 01:41:08.492568 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:08.492539 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fdbeda1-eb2d-400f-a2d9-82b28334ca60-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w\" (UID: \"7fdbeda1-eb2d-400f-a2d9-82b28334ca60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w" Nov 04 01:41:08.494448 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:08.494421 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fdbeda1-eb2d-400f-a2d9-82b28334ca60-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w\" (UID: \"7fdbeda1-eb2d-400f-a2d9-82b28334ca60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w" Nov 04 01:41:08.494598 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:08.494575 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fdbeda1-eb2d-400f-a2d9-82b28334ca60-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w\" (UID: \"7fdbeda1-eb2d-400f-a2d9-82b28334ca60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w" Nov 04 01:41:08.502241 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:08.502212 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dcx9\" (UniqueName: \"kubernetes.io/projected/7fdbeda1-eb2d-400f-a2d9-82b28334ca60-kube-api-access-4dcx9\") pod \"maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w\" (UID: \"7fdbeda1-eb2d-400f-a2d9-82b28334ca60\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w" Nov 04 01:41:08.657528 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:08.657447 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w" Nov 04 01:41:08.939826 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:08.939795 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w"] Nov 04 01:41:08.941369 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:41:08.941344 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fdbeda1_eb2d_400f_a2d9_82b28334ca60.slice/crio-68f18248f34b1ce66875a91dd144a6e106fb78730ecd917febd2b233ce1a12bb WatchSource:0}: Error finding container 68f18248f34b1ce66875a91dd144a6e106fb78730ecd917febd2b233ce1a12bb: Status 404 returned error can't find the container with id 68f18248f34b1ce66875a91dd144a6e106fb78730ecd917febd2b233ce1a12bb Nov 04 01:41:08.942894 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:08.942876 7059 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:41:09.880933 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:09.880877 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w" event={"ID":"7fdbeda1-eb2d-400f-a2d9-82b28334ca60","Type":"ContainerStarted","Data":"cdae9e4100843fb693037381b2d44b13bf1a645b2eb1c971a85f993ec86f0894"} Nov 04 01:41:09.881323 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:09.880954 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w" event={"ID":"7fdbeda1-eb2d-400f-a2d9-82b28334ca60","Type":"ContainerStarted","Data":"68f18248f34b1ce66875a91dd144a6e106fb78730ecd917febd2b233ce1a12bb"} Nov 04 01:41:09.881323 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:09.881124 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w" Nov 04 01:41:09.901490 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:09.901445 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w" podStartSLOduration=1.456244205 podStartE2EDuration="1.90143253s" totalImagesPullingTime="445.188325ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:41:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:41:09.897470969 +0000 UTC m=+2554.839319874" watchObservedRunningTime="2026-11-04 01:41:09.90143253 +0000 UTC m=+2554.843281411" Nov 04 01:41:15.893077 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:15.893048 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w" Nov 04 01:41:24.389990 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:24.389957 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h"] Nov 04 01:41:24.394265 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:24.394242 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h" Nov 04 01:41:24.401717 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:24.401694 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-b9b238\"/\"default-dockercfg-vvhzg\"" Nov 04 01:41:24.401895 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:24.401749 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-b9b238\"/\"simulator-single-main-b9b238-kserve-self-signed-certs\"" Nov 04 01:41:24.405355 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:24.405328 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h"] Nov 04 01:41:24.469379 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:24.469348 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-b9b238\"/\"kube-root-ca.crt\"" Nov 04 01:41:24.469569 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:24.469457 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-b9b238\"/\"openshift-service-ca.crt\"" Nov 04 01:41:24.556870 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:24.556840 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23ca1a0d-8310-479c-9156-ec74ac0a64fe-dshm\") pod \"simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h\" (UID: \"23ca1a0d-8310-479c-9156-ec74ac0a64fe\") " pod="e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h" Nov 04 01:41:24.556870 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:24.556878 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23ca1a0d-8310-479c-9156-ec74ac0a64fe-home\") pod \"simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h\" (UID: \"23ca1a0d-8310-479c-9156-ec74ac0a64fe\") " pod="e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h" Nov 04 01:41:24.557573 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:24.556908 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23ca1a0d-8310-479c-9156-ec74ac0a64fe-tmp-dir\") pod \"simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h\" (UID: \"23ca1a0d-8310-479c-9156-ec74ac0a64fe\") " pod="e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h" Nov 04 01:41:24.557573 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:24.557013 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23ca1a0d-8310-479c-9156-ec74ac0a64fe-model-cache\") pod \"simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h\" (UID: \"23ca1a0d-8310-479c-9156-ec74ac0a64fe\") " pod="e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h" Nov 04 01:41:24.557573 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:24.557156 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5gg6\" (UniqueName: \"kubernetes.io/projected/23ca1a0d-8310-479c-9156-ec74ac0a64fe-kube-api-access-l5gg6\") pod \"simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h\" (UID: \"23ca1a0d-8310-479c-9156-ec74ac0a64fe\") " pod="e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h" Nov 04 01:41:24.557573 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:24.557292 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23ca1a0d-8310-479c-9156-ec74ac0a64fe-tls-certs\") pod \"simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h\" (UID: \"23ca1a0d-8310-479c-9156-ec74ac0a64fe\") " pod="e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h" Nov 04 01:41:24.658539 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:24.658452 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23ca1a0d-8310-479c-9156-ec74ac0a64fe-tls-certs\") pod \"simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h\" (UID: \"23ca1a0d-8310-479c-9156-ec74ac0a64fe\") " pod="e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h" Nov 04 01:41:24.658539 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:24.658489 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23ca1a0d-8310-479c-9156-ec74ac0a64fe-dshm\") pod \"simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h\" (UID: \"23ca1a0d-8310-479c-9156-ec74ac0a64fe\") " pod="e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h" Nov 04 01:41:24.658539 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:24.658515 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23ca1a0d-8310-479c-9156-ec74ac0a64fe-home\") pod \"simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h\" (UID: \"23ca1a0d-8310-479c-9156-ec74ac0a64fe\") " pod="e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h" Nov 04 01:41:24.658839 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:24.658549 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23ca1a0d-8310-479c-9156-ec74ac0a64fe-tmp-dir\") pod \"simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h\" (UID: \"23ca1a0d-8310-479c-9156-ec74ac0a64fe\") " pod="e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h" Nov 04 01:41:24.658839 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:24.658577 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23ca1a0d-8310-479c-9156-ec74ac0a64fe-model-cache\") pod \"simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h\" (UID: \"23ca1a0d-8310-479c-9156-ec74ac0a64fe\") " pod="e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h" Nov 04 01:41:24.658839 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:24.658678 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5gg6\" (UniqueName: \"kubernetes.io/projected/23ca1a0d-8310-479c-9156-ec74ac0a64fe-kube-api-access-l5gg6\") pod \"simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h\" (UID: \"23ca1a0d-8310-479c-9156-ec74ac0a64fe\") " pod="e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h" Nov 04 01:41:24.659126 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:24.659094 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23ca1a0d-8310-479c-9156-ec74ac0a64fe-model-cache\") pod \"simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h\" (UID: \"23ca1a0d-8310-479c-9156-ec74ac0a64fe\") " pod="e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h" Nov 04 01:41:24.659251 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:24.659099 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23ca1a0d-8310-479c-9156-ec74ac0a64fe-tmp-dir\") pod \"simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h\" (UID: \"23ca1a0d-8310-479c-9156-ec74ac0a64fe\") " pod="e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h" Nov 04 01:41:24.659467 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:24.659444 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23ca1a0d-8310-479c-9156-ec74ac0a64fe-home\") pod \"simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h\" (UID: \"23ca1a0d-8310-479c-9156-ec74ac0a64fe\") " pod="e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h" Nov 04 01:41:24.660974 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:24.660955 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23ca1a0d-8310-479c-9156-ec74ac0a64fe-dshm\") pod \"simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h\" (UID: \"23ca1a0d-8310-479c-9156-ec74ac0a64fe\") " pod="e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h" Nov 04 01:41:24.661619 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:24.661599 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23ca1a0d-8310-479c-9156-ec74ac0a64fe-tls-certs\") pod \"simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h\" (UID: \"23ca1a0d-8310-479c-9156-ec74ac0a64fe\") " pod="e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h" Nov 04 01:41:24.670529 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:24.670509 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5gg6\" (UniqueName: \"kubernetes.io/projected/23ca1a0d-8310-479c-9156-ec74ac0a64fe-kube-api-access-l5gg6\") pod \"simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h\" (UID: \"23ca1a0d-8310-479c-9156-ec74ac0a64fe\") " pod="e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h" Nov 04 01:41:24.712326 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:24.712301 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h" Nov 04 01:41:24.996739 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:24.996702 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h"] Nov 04 01:41:24.998161 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:41:24.998136 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23ca1a0d_8310_479c_9156_ec74ac0a64fe.slice/crio-6e6ed86ba5c299e8e62a8f85028203f95e9890a7db0cf6d403dfc2368a683936 WatchSource:0}: Error finding container 6e6ed86ba5c299e8e62a8f85028203f95e9890a7db0cf6d403dfc2368a683936: Status 404 returned error can't find the container with id 6e6ed86ba5c299e8e62a8f85028203f95e9890a7db0cf6d403dfc2368a683936 Nov 04 01:41:25.981242 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:25.981201 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h" event={"ID":"23ca1a0d-8310-479c-9156-ec74ac0a64fe","Type":"ContainerStarted","Data":"82f7fb750d26e932ccd347b3eb9e31d4080cb4953f172ab0721af11d32c69d32"} Nov 04 01:41:25.981242 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:25.981245 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h" event={"ID":"23ca1a0d-8310-479c-9156-ec74ac0a64fe","Type":"ContainerStarted","Data":"6e6ed86ba5c299e8e62a8f85028203f95e9890a7db0cf6d403dfc2368a683936"} Nov 04 01:41:25.999344 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:25.999296 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h" podStartSLOduration=1.9992831089999998 podStartE2EDuration="1.999283109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:41:25.997834126 +0000 UTC m=+2570.939683030" watchObservedRunningTime="2026-11-04 01:41:25.999283109 +0000 UTC m=+2570.941132011" Nov 04 01:41:26.713426 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:26.713398 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h" Nov 04 01:41:32.026019 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:32.025986 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4cprq"] Nov 04 01:41:32.027893 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:32.027871 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4cprq"] Nov 04 01:41:33.175973 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:33.175919 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88c9538e-6964-4310-839a-9d8c2774501f" path="/var/lib/kubelet/pods/88c9538e-6964-4310-839a-9d8c2774501f/volumes" Nov 04 01:41:34.712846 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:34.712809 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h" Nov 04 01:41:34.725862 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:34.725833 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h" Nov 04 01:41:35.054179 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:41:35.054102 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h" Nov 04 01:41:36.480177 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:41:36.480136 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:42:06.480836 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:42:06.480801 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:42:09.653357 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:09.653322 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h"] Nov 04 01:42:11.870693 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:11.870662 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w"] Nov 04 01:42:11.872028 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:11.871992 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w" podUID="7fdbeda1-eb2d-400f-a2d9-82b28334ca60" containerName="maas-api" containerID="cri-o://cdae9e4100843fb693037381b2d44b13bf1a645b2eb1c971a85f993ec86f0894" gracePeriod=30 Nov 04 01:42:12.143430 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:12.143406 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w" Nov 04 01:42:12.240219 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:12.240188 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fdbeda1-eb2d-400f-a2d9-82b28334ca60-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fdbeda1-eb2d-400f-a2d9-82b28334ca60-maas-api-tls\") pod \"7fdbeda1-eb2d-400f-a2d9-82b28334ca60\" (UID: \"7fdbeda1-eb2d-400f-a2d9-82b28334ca60\") " Nov 04 01:42:12.240390 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:12.240237 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fdbeda1-eb2d-400f-a2d9-82b28334ca60-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fdbeda1-eb2d-400f-a2d9-82b28334ca60-metrics-certs\") pod \"7fdbeda1-eb2d-400f-a2d9-82b28334ca60\" (UID: \"7fdbeda1-eb2d-400f-a2d9-82b28334ca60\") " Nov 04 01:42:12.240390 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:12.240282 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fdbeda1-eb2d-400f-a2d9-82b28334ca60-kube-api-access-4dcx9\" (UniqueName: \"kubernetes.io/projected/7fdbeda1-eb2d-400f-a2d9-82b28334ca60-kube-api-access-4dcx9\") pod \"7fdbeda1-eb2d-400f-a2d9-82b28334ca60\" (UID: \"7fdbeda1-eb2d-400f-a2d9-82b28334ca60\") " Nov 04 01:42:12.240390 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:12.240316 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fdbeda1-eb2d-400f-a2d9-82b28334ca60-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fdbeda1-eb2d-400f-a2d9-82b28334ca60-openshift-service-ca\") pod \"7fdbeda1-eb2d-400f-a2d9-82b28334ca60\" (UID: \"7fdbeda1-eb2d-400f-a2d9-82b28334ca60\") " Nov 04 01:42:12.240753 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:12.240718 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fdbeda1-eb2d-400f-a2d9-82b28334ca60-openshift-service-ca" pod "7fdbeda1-eb2d-400f-a2d9-82b28334ca60" (UID: "7fdbeda1-eb2d-400f-a2d9-82b28334ca60"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:42:12.242540 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:12.242512 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fdbeda1-eb2d-400f-a2d9-82b28334ca60-maas-api-tls" pod "7fdbeda1-eb2d-400f-a2d9-82b28334ca60" (UID: "7fdbeda1-eb2d-400f-a2d9-82b28334ca60"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:42:12.242634 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:12.242554 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fdbeda1-eb2d-400f-a2d9-82b28334ca60-kube-api-access-4dcx9" pod "7fdbeda1-eb2d-400f-a2d9-82b28334ca60" (UID: "7fdbeda1-eb2d-400f-a2d9-82b28334ca60"). InnerVolumeSpecName "kube-api-access-4dcx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:42:12.242693 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:12.242632 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fdbeda1-eb2d-400f-a2d9-82b28334ca60-metrics-certs" pod "7fdbeda1-eb2d-400f-a2d9-82b28334ca60" (UID: "7fdbeda1-eb2d-400f-a2d9-82b28334ca60"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:42:12.281124 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:12.281089 7059 generic.go:356] "Generic (PLEG): container finished" podID="7fdbeda1-eb2d-400f-a2d9-82b28334ca60" containerID="cdae9e4100843fb693037381b2d44b13bf1a645b2eb1c971a85f993ec86f0894" exitCode=0 Nov 04 01:42:12.281255 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:12.281147 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w" Nov 04 01:42:12.281255 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:12.281174 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w" event={"ID":"7fdbeda1-eb2d-400f-a2d9-82b28334ca60","Type":"ContainerDied","Data":"cdae9e4100843fb693037381b2d44b13bf1a645b2eb1c971a85f993ec86f0894"} Nov 04 01:42:12.281255 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:12.281239 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w" event={"ID":"7fdbeda1-eb2d-400f-a2d9-82b28334ca60","Type":"ContainerDied","Data":"68f18248f34b1ce66875a91dd144a6e106fb78730ecd917febd2b233ce1a12bb"} Nov 04 01:42:12.281427 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:12.281271 7059 scope.go:122] "RemoveContainer" containerID="cdae9e4100843fb693037381b2d44b13bf1a645b2eb1c971a85f993ec86f0894" Nov 04 01:42:12.292185 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:12.292168 7059 scope.go:122] "RemoveContainer" containerID="cdae9e4100843fb693037381b2d44b13bf1a645b2eb1c971a85f993ec86f0894" Nov 04 01:42:12.292450 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:42:12.292430 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdae9e4100843fb693037381b2d44b13bf1a645b2eb1c971a85f993ec86f0894\": container with ID starting with cdae9e4100843fb693037381b2d44b13bf1a645b2eb1c971a85f993ec86f0894 not found: ID does not exist" containerID="cdae9e4100843fb693037381b2d44b13bf1a645b2eb1c971a85f993ec86f0894" Nov 04 01:42:12.292514 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:12.292457 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdae9e4100843fb693037381b2d44b13bf1a645b2eb1c971a85f993ec86f0894"} err="failed to get container status \"cdae9e4100843fb693037381b2d44b13bf1a645b2eb1c971a85f993ec86f0894\": rpc error: code = NotFound desc = could not find container \"cdae9e4100843fb693037381b2d44b13bf1a645b2eb1c971a85f993ec86f0894\": container with ID starting with cdae9e4100843fb693037381b2d44b13bf1a645b2eb1c971a85f993ec86f0894 not found: ID does not exist" Nov 04 01:42:12.304217 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:12.304190 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w"] Nov 04 01:42:12.310116 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:12.310095 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-9dr7w"] Nov 04 01:42:12.341350 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:12.341329 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dcx9\" (UniqueName: \"kubernetes.io/projected/7fdbeda1-eb2d-400f-a2d9-82b28334ca60-kube-api-access-4dcx9\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:42:12.341439 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:12.341353 7059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fdbeda1-eb2d-400f-a2d9-82b28334ca60-openshift-service-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:42:12.341439 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:12.341364 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fdbeda1-eb2d-400f-a2d9-82b28334ca60-maas-api-tls\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:42:12.341439 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:12.341374 7059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fdbeda1-eb2d-400f-a2d9-82b28334ca60-metrics-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:42:13.176224 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:13.176181 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fdbeda1-eb2d-400f-a2d9-82b28334ca60" path="/var/lib/kubelet/pods/7fdbeda1-eb2d-400f-a2d9-82b28334ca60/volumes" Nov 04 01:42:23.053211 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:23.053130 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-b94h2"] Nov 04 01:42:23.064818 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:23.064782 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-b94h2"] Nov 04 01:42:23.177170 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:23.177135 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bba91f06-160f-4e53-a1f4-16c4b390db33" path="/var/lib/kubelet/pods/bba91f06-160f-4e53-a1f4-16c4b390db33/volumes" Nov 04 01:42:24.674854 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:24.674816 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h" podUID="23ca1a0d-8310-479c-9156-ec74ac0a64fe" containerName="main" containerID="cri-o://82f7fb750d26e932ccd347b3eb9e31d4080cb4953f172ab0721af11d32c69d32" gracePeriod=45 Nov 04 01:42:24.934499 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:24.934437 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h" Nov 04 01:42:25.057793 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:25.057754 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23ca1a0d-8310-479c-9156-ec74ac0a64fe-tls-certs\" (UniqueName: \"kubernetes.io/secret/23ca1a0d-8310-479c-9156-ec74ac0a64fe-tls-certs\") pod \"23ca1a0d-8310-479c-9156-ec74ac0a64fe\" (UID: \"23ca1a0d-8310-479c-9156-ec74ac0a64fe\") " Nov 04 01:42:25.058010 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:25.057803 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ca1a0d-8310-479c-9156-ec74ac0a64fe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23ca1a0d-8310-479c-9156-ec74ac0a64fe-tmp-dir\") pod \"23ca1a0d-8310-479c-9156-ec74ac0a64fe\" (UID: \"23ca1a0d-8310-479c-9156-ec74ac0a64fe\") " Nov 04 01:42:25.058010 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:25.057847 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ca1a0d-8310-479c-9156-ec74ac0a64fe-home\" (UniqueName: \"kubernetes.io/empty-dir/23ca1a0d-8310-479c-9156-ec74ac0a64fe-home\") pod \"23ca1a0d-8310-479c-9156-ec74ac0a64fe\" (UID: \"23ca1a0d-8310-479c-9156-ec74ac0a64fe\") " Nov 04 01:42:25.058010 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:25.057894 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ca1a0d-8310-479c-9156-ec74ac0a64fe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23ca1a0d-8310-479c-9156-ec74ac0a64fe-model-cache\") pod \"23ca1a0d-8310-479c-9156-ec74ac0a64fe\" (UID: \"23ca1a0d-8310-479c-9156-ec74ac0a64fe\") " Nov 04 01:42:25.058010 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:25.057960 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23ca1a0d-8310-479c-9156-ec74ac0a64fe-kube-api-access-l5gg6\" (UniqueName: \"kubernetes.io/projected/23ca1a0d-8310-479c-9156-ec74ac0a64fe-kube-api-access-l5gg6\") pod \"23ca1a0d-8310-479c-9156-ec74ac0a64fe\" (UID: \"23ca1a0d-8310-479c-9156-ec74ac0a64fe\") " Nov 04 01:42:25.058010 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:25.057996 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ca1a0d-8310-479c-9156-ec74ac0a64fe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/23ca1a0d-8310-479c-9156-ec74ac0a64fe-dshm\") pod \"23ca1a0d-8310-479c-9156-ec74ac0a64fe\" (UID: \"23ca1a0d-8310-479c-9156-ec74ac0a64fe\") " Nov 04 01:42:25.058269 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:25.058132 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ca1a0d-8310-479c-9156-ec74ac0a64fe-home" pod "23ca1a0d-8310-479c-9156-ec74ac0a64fe" (UID: "23ca1a0d-8310-479c-9156-ec74ac0a64fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:42:25.058269 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:25.058148 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ca1a0d-8310-479c-9156-ec74ac0a64fe-model-cache" pod "23ca1a0d-8310-479c-9156-ec74ac0a64fe" (UID: "23ca1a0d-8310-479c-9156-ec74ac0a64fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:42:25.058269 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:25.058159 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ca1a0d-8310-479c-9156-ec74ac0a64fe-tmp-dir" pod "23ca1a0d-8310-479c-9156-ec74ac0a64fe" (UID: "23ca1a0d-8310-479c-9156-ec74ac0a64fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:42:25.058434 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:25.058417 7059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23ca1a0d-8310-479c-9156-ec74ac0a64fe-tmp-dir\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:42:25.058471 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:25.058437 7059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23ca1a0d-8310-479c-9156-ec74ac0a64fe-home\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:42:25.058471 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:25.058448 7059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23ca1a0d-8310-479c-9156-ec74ac0a64fe-model-cache\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:42:25.061607 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:25.061547 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ca1a0d-8310-479c-9156-ec74ac0a64fe-kube-api-access-l5gg6" pod "23ca1a0d-8310-479c-9156-ec74ac0a64fe" (UID: "23ca1a0d-8310-479c-9156-ec74ac0a64fe"). InnerVolumeSpecName "kube-api-access-l5gg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:42:25.061607 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:25.061551 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ca1a0d-8310-479c-9156-ec74ac0a64fe-dshm" pod "23ca1a0d-8310-479c-9156-ec74ac0a64fe" (UID: "23ca1a0d-8310-479c-9156-ec74ac0a64fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:42:25.062017 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:25.061625 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23ca1a0d-8310-479c-9156-ec74ac0a64fe-tls-certs" pod "23ca1a0d-8310-479c-9156-ec74ac0a64fe" (UID: "23ca1a0d-8310-479c-9156-ec74ac0a64fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:42:25.159054 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:25.159026 7059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23ca1a0d-8310-479c-9156-ec74ac0a64fe-tls-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:42:25.159159 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:25.159053 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5gg6\" (UniqueName: \"kubernetes.io/projected/23ca1a0d-8310-479c-9156-ec74ac0a64fe-kube-api-access-l5gg6\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:42:25.159159 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:25.159069 7059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23ca1a0d-8310-479c-9156-ec74ac0a64fe-dshm\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:42:25.357459 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:25.357392 7059 generic.go:356] "Generic (PLEG): container finished" podID="23ca1a0d-8310-479c-9156-ec74ac0a64fe" containerID="82f7fb750d26e932ccd347b3eb9e31d4080cb4953f172ab0721af11d32c69d32" exitCode=0 Nov 04 01:42:25.357459 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:25.357437 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h" Nov 04 01:42:25.357625 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:25.357480 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h" event={"ID":"23ca1a0d-8310-479c-9156-ec74ac0a64fe","Type":"ContainerDied","Data":"82f7fb750d26e932ccd347b3eb9e31d4080cb4953f172ab0721af11d32c69d32"} Nov 04 01:42:25.357625 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:25.357525 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h" event={"ID":"23ca1a0d-8310-479c-9156-ec74ac0a64fe","Type":"ContainerDied","Data":"6e6ed86ba5c299e8e62a8f85028203f95e9890a7db0cf6d403dfc2368a683936"} Nov 04 01:42:25.357625 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:25.357549 7059 scope.go:122] "RemoveContainer" containerID="82f7fb750d26e932ccd347b3eb9e31d4080cb4953f172ab0721af11d32c69d32" Nov 04 01:42:25.367462 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:25.367440 7059 scope.go:122] "RemoveContainer" containerID="82f7fb750d26e932ccd347b3eb9e31d4080cb4953f172ab0721af11d32c69d32" Nov 04 01:42:25.367755 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:42:25.367733 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82f7fb750d26e932ccd347b3eb9e31d4080cb4953f172ab0721af11d32c69d32\": container with ID starting with 82f7fb750d26e932ccd347b3eb9e31d4080cb4953f172ab0721af11d32c69d32 not found: ID does not exist" containerID="82f7fb750d26e932ccd347b3eb9e31d4080cb4953f172ab0721af11d32c69d32" Nov 04 01:42:25.367796 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:25.367767 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82f7fb750d26e932ccd347b3eb9e31d4080cb4953f172ab0721af11d32c69d32"} err="failed to get container status \"82f7fb750d26e932ccd347b3eb9e31d4080cb4953f172ab0721af11d32c69d32\": rpc error: code = NotFound desc = could not find container \"82f7fb750d26e932ccd347b3eb9e31d4080cb4953f172ab0721af11d32c69d32\": container with ID starting with 82f7fb750d26e932ccd347b3eb9e31d4080cb4953f172ab0721af11d32c69d32 not found: ID does not exist" Nov 04 01:42:25.375439 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:25.375418 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h"] Nov 04 01:42:25.381009 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:25.380988 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-b9b238/simulator-single-main-b9b238-kserve-66f4fc89b4-frr9h"] Nov 04 01:42:27.176038 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:27.176005 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23ca1a0d-8310-479c-9156-ec74ac0a64fe" path="/var/lib/kubelet/pods/23ca1a0d-8310-479c-9156-ec74ac0a64fe/volumes" Nov 04 01:42:36.450366 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:36.450332 7059 scope.go:122] "RemoveContainer" containerID="f8f75d5982f067a50ce6c6dd1725127d7f1accd3e6d4505dc9fac06ff35423b1" Nov 04 01:42:36.479997 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:42:36.479965 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:42:40.023053 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:40.023020 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m2z56"] Nov 04 01:42:40.027510 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:40.027482 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m2z56"] Nov 04 01:42:41.175678 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:41.175643 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a40f207-413e-48d0-b971-f350aae4bbd6" path="/var/lib/kubelet/pods/2a40f207-413e-48d0-b971-f350aae4bbd6/volumes" Nov 04 01:42:43.430224 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:43.430191 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c"] Nov 04 01:42:43.430649 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:43.430629 7059 cpu_manager.go:414] "containerMap: removing container" podUID="7fdbeda1-eb2d-400f-a2d9-82b28334ca60" containerName="maas-api" Nov 04 01:42:43.430697 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:43.430649 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fdbeda1-eb2d-400f-a2d9-82b28334ca60" containerName="maas-api" Nov 04 01:42:43.430697 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:43.430670 7059 cpu_manager.go:414] "containerMap: removing container" podUID="23ca1a0d-8310-479c-9156-ec74ac0a64fe" containerName="main" Nov 04 01:42:43.430697 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:43.430675 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23ca1a0d-8310-479c-9156-ec74ac0a64fe" containerName="main" Nov 04 01:42:43.430794 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:43.430753 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="23ca1a0d-8310-479c-9156-ec74ac0a64fe" containerName="main" Nov 04 01:42:43.430794 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:43.430767 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fdbeda1-eb2d-400f-a2d9-82b28334ca60" containerName="maas-api" Nov 04 01:42:43.435606 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:43.435587 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c" Nov 04 01:42:43.444203 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:43.444179 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nb79d\"" Nov 04 01:42:43.445015 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:43.444996 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-b9b238-serving-cert\"" Nov 04 01:42:43.448780 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:43.448761 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 04 01:42:43.459111 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:43.459089 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c"] Nov 04 01:42:43.523093 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:43.523065 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8532c13f-1f8b-4e27-97c6-48f82e7c7bdc-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c\" (UID: \"8532c13f-1f8b-4e27-97c6-48f82e7c7bdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c" Nov 04 01:42:43.523254 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:43.523201 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8532c13f-1f8b-4e27-97c6-48f82e7c7bdc-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c\" (UID: \"8532c13f-1f8b-4e27-97c6-48f82e7c7bdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c" Nov 04 01:42:43.523254 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:43.523238 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8532c13f-1f8b-4e27-97c6-48f82e7c7bdc-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c\" (UID: \"8532c13f-1f8b-4e27-97c6-48f82e7c7bdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c" Nov 04 01:42:43.523346 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:43.523311 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55w5x\" (UniqueName: \"kubernetes.io/projected/8532c13f-1f8b-4e27-97c6-48f82e7c7bdc-kube-api-access-55w5x\") pod \"maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c\" (UID: \"8532c13f-1f8b-4e27-97c6-48f82e7c7bdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c" Nov 04 01:42:43.624338 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:43.624306 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8532c13f-1f8b-4e27-97c6-48f82e7c7bdc-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c\" (UID: \"8532c13f-1f8b-4e27-97c6-48f82e7c7bdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c" Nov 04 01:42:43.624490 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:43.624386 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8532c13f-1f8b-4e27-97c6-48f82e7c7bdc-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c\" (UID: \"8532c13f-1f8b-4e27-97c6-48f82e7c7bdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c" Nov 04 01:42:43.624490 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:43.624402 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8532c13f-1f8b-4e27-97c6-48f82e7c7bdc-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c\" (UID: \"8532c13f-1f8b-4e27-97c6-48f82e7c7bdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c" Nov 04 01:42:43.624490 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:43.624433 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55w5x\" (UniqueName: \"kubernetes.io/projected/8532c13f-1f8b-4e27-97c6-48f82e7c7bdc-kube-api-access-55w5x\") pod \"maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c\" (UID: \"8532c13f-1f8b-4e27-97c6-48f82e7c7bdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c" Nov 04 01:42:43.624992 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:43.624969 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8532c13f-1f8b-4e27-97c6-48f82e7c7bdc-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c\" (UID: \"8532c13f-1f8b-4e27-97c6-48f82e7c7bdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c" Nov 04 01:42:43.627169 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:43.627145 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8532c13f-1f8b-4e27-97c6-48f82e7c7bdc-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c\" (UID: \"8532c13f-1f8b-4e27-97c6-48f82e7c7bdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c" Nov 04 01:42:43.627271 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:43.627191 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8532c13f-1f8b-4e27-97c6-48f82e7c7bdc-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c\" (UID: \"8532c13f-1f8b-4e27-97c6-48f82e7c7bdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c" Nov 04 01:42:43.634082 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:43.634047 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55w5x\" (UniqueName: \"kubernetes.io/projected/8532c13f-1f8b-4e27-97c6-48f82e7c7bdc-kube-api-access-55w5x\") pod \"maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c\" (UID: \"8532c13f-1f8b-4e27-97c6-48f82e7c7bdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c" Nov 04 01:42:43.749189 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:43.749101 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c" Nov 04 01:42:44.034877 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:44.034398 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c"] Nov 04 01:42:44.036452 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:42:44.036410 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8532c13f_1f8b_4e27_97c6_48f82e7c7bdc.slice/crio-919f8d1b312c0758265871dbc122467c80311a5e25f57d8cf4f37231604dc72b WatchSource:0}: Error finding container 919f8d1b312c0758265871dbc122467c80311a5e25f57d8cf4f37231604dc72b: Status 404 returned error can't find the container with id 919f8d1b312c0758265871dbc122467c80311a5e25f57d8cf4f37231604dc72b Nov 04 01:42:44.476525 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:44.476490 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c" event={"ID":"8532c13f-1f8b-4e27-97c6-48f82e7c7bdc","Type":"ContainerStarted","Data":"919f8d1b312c0758265871dbc122467c80311a5e25f57d8cf4f37231604dc72b"} Nov 04 01:42:45.484395 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:45.484352 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c" event={"ID":"8532c13f-1f8b-4e27-97c6-48f82e7c7bdc","Type":"ContainerStarted","Data":"5599d1c2af268603d1cb083b1ffed33594574ca0b130ec85f2bac9284983ab37"} Nov 04 01:42:45.484803 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:45.484431 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c" Nov 04 01:42:45.503525 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:45.503480 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c" podStartSLOduration=1.5830501300000002 podStartE2EDuration="2.503464828s" totalImagesPullingTime="920.414698ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:42:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:42:45.501641027 +0000 UTC m=+2650.443489930" watchObservedRunningTime="2026-11-04 01:42:45.503464828 +0000 UTC m=+2650.445313731" Nov 04 01:42:51.021956 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:51.021902 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jfz9c"] Nov 04 01:42:51.025578 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:51.025540 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jfz9c"] Nov 04 01:42:51.177128 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:51.177095 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c3ef6ac-14cf-4cd6-a2de-9d241e055e71" path="/var/lib/kubelet/pods/5c3ef6ac-14cf-4cd6-a2de-9d241e055e71/volumes" Nov 04 01:42:51.498336 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:42:51.498309 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c" Nov 04 01:43:03.137246 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:03.137203 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c"] Nov 04 01:43:03.137697 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:03.137462 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c" podUID="8532c13f-1f8b-4e27-97c6-48f82e7c7bdc" containerName="maas-api" containerID="cri-o://5599d1c2af268603d1cb083b1ffed33594574ca0b130ec85f2bac9284983ab37" gracePeriod=30 Nov 04 01:43:03.409501 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:03.409475 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c" Nov 04 01:43:03.423305 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:03.423278 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8532c13f-1f8b-4e27-97c6-48f82e7c7bdc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8532c13f-1f8b-4e27-97c6-48f82e7c7bdc-openshift-service-ca\") pod \"8532c13f-1f8b-4e27-97c6-48f82e7c7bdc\" (UID: \"8532c13f-1f8b-4e27-97c6-48f82e7c7bdc\") " Nov 04 01:43:03.423421 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:03.423390 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8532c13f-1f8b-4e27-97c6-48f82e7c7bdc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8532c13f-1f8b-4e27-97c6-48f82e7c7bdc-maas-api-tls\") pod \"8532c13f-1f8b-4e27-97c6-48f82e7c7bdc\" (UID: \"8532c13f-1f8b-4e27-97c6-48f82e7c7bdc\") " Nov 04 01:43:03.423468 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:03.423457 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8532c13f-1f8b-4e27-97c6-48f82e7c7bdc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8532c13f-1f8b-4e27-97c6-48f82e7c7bdc-metrics-certs\") pod \"8532c13f-1f8b-4e27-97c6-48f82e7c7bdc\" (UID: \"8532c13f-1f8b-4e27-97c6-48f82e7c7bdc\") " Nov 04 01:43:03.423578 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:03.423561 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8532c13f-1f8b-4e27-97c6-48f82e7c7bdc-kube-api-access-55w5x\" (UniqueName: \"kubernetes.io/projected/8532c13f-1f8b-4e27-97c6-48f82e7c7bdc-kube-api-access-55w5x\") pod \"8532c13f-1f8b-4e27-97c6-48f82e7c7bdc\" (UID: \"8532c13f-1f8b-4e27-97c6-48f82e7c7bdc\") " Nov 04 01:43:03.423662 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:03.423640 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8532c13f-1f8b-4e27-97c6-48f82e7c7bdc-openshift-service-ca" pod "8532c13f-1f8b-4e27-97c6-48f82e7c7bdc" (UID: "8532c13f-1f8b-4e27-97c6-48f82e7c7bdc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:43:03.423900 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:03.423884 7059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8532c13f-1f8b-4e27-97c6-48f82e7c7bdc-openshift-service-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:03.425631 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:03.425605 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8532c13f-1f8b-4e27-97c6-48f82e7c7bdc-maas-api-tls" pod "8532c13f-1f8b-4e27-97c6-48f82e7c7bdc" (UID: "8532c13f-1f8b-4e27-97c6-48f82e7c7bdc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:43:03.425708 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:03.425625 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8532c13f-1f8b-4e27-97c6-48f82e7c7bdc-metrics-certs" pod "8532c13f-1f8b-4e27-97c6-48f82e7c7bdc" (UID: "8532c13f-1f8b-4e27-97c6-48f82e7c7bdc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:43:03.425769 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:03.425754 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8532c13f-1f8b-4e27-97c6-48f82e7c7bdc-kube-api-access-55w5x" pod "8532c13f-1f8b-4e27-97c6-48f82e7c7bdc" (UID: "8532c13f-1f8b-4e27-97c6-48f82e7c7bdc"). InnerVolumeSpecName "kube-api-access-55w5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:43:03.524846 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:03.524814 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55w5x\" (UniqueName: \"kubernetes.io/projected/8532c13f-1f8b-4e27-97c6-48f82e7c7bdc-kube-api-access-55w5x\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:03.524846 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:03.524841 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8532c13f-1f8b-4e27-97c6-48f82e7c7bdc-maas-api-tls\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:03.524846 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:03.524851 7059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8532c13f-1f8b-4e27-97c6-48f82e7c7bdc-metrics-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:03.597279 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:03.597245 7059 generic.go:356] "Generic (PLEG): container finished" podID="8532c13f-1f8b-4e27-97c6-48f82e7c7bdc" containerID="5599d1c2af268603d1cb083b1ffed33594574ca0b130ec85f2bac9284983ab37" exitCode=0 Nov 04 01:43:03.597454 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:03.597298 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c" Nov 04 01:43:03.597454 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:03.597335 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c" event={"ID":"8532c13f-1f8b-4e27-97c6-48f82e7c7bdc","Type":"ContainerDied","Data":"5599d1c2af268603d1cb083b1ffed33594574ca0b130ec85f2bac9284983ab37"} Nov 04 01:43:03.597454 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:03.597389 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c" event={"ID":"8532c13f-1f8b-4e27-97c6-48f82e7c7bdc","Type":"ContainerDied","Data":"919f8d1b312c0758265871dbc122467c80311a5e25f57d8cf4f37231604dc72b"} Nov 04 01:43:03.597454 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:03.597411 7059 scope.go:122] "RemoveContainer" containerID="5599d1c2af268603d1cb083b1ffed33594574ca0b130ec85f2bac9284983ab37" Nov 04 01:43:03.608548 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:03.608529 7059 scope.go:122] "RemoveContainer" containerID="5599d1c2af268603d1cb083b1ffed33594574ca0b130ec85f2bac9284983ab37" Nov 04 01:43:03.608814 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:43:03.608792 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5599d1c2af268603d1cb083b1ffed33594574ca0b130ec85f2bac9284983ab37\": container with ID starting with 5599d1c2af268603d1cb083b1ffed33594574ca0b130ec85f2bac9284983ab37 not found: ID does not exist" containerID="5599d1c2af268603d1cb083b1ffed33594574ca0b130ec85f2bac9284983ab37" Nov 04 01:43:03.608900 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:03.608824 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5599d1c2af268603d1cb083b1ffed33594574ca0b130ec85f2bac9284983ab37"} err="failed to get container status \"5599d1c2af268603d1cb083b1ffed33594574ca0b130ec85f2bac9284983ab37\": rpc error: code = NotFound desc = could not find container \"5599d1c2af268603d1cb083b1ffed33594574ca0b130ec85f2bac9284983ab37\": container with ID starting with 5599d1c2af268603d1cb083b1ffed33594574ca0b130ec85f2bac9284983ab37 not found: ID does not exist" Nov 04 01:43:03.620053 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:03.620030 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c"] Nov 04 01:43:03.624445 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:03.624422 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9b238-5b8b6556c4-6zl4c"] Nov 04 01:43:05.029984 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:05.029952 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vj66h"] Nov 04 01:43:05.033855 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:05.033823 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vj66h"] Nov 04 01:43:05.048084 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:05.048061 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzrrd"] Nov 04 01:43:05.048574 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:05.048560 7059 cpu_manager.go:414] "containerMap: removing container" podUID="8532c13f-1f8b-4e27-97c6-48f82e7c7bdc" containerName="maas-api" Nov 04 01:43:05.048630 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:05.048575 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8532c13f-1f8b-4e27-97c6-48f82e7c7bdc" containerName="maas-api" Nov 04 01:43:05.048666 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:05.048655 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8532c13f-1f8b-4e27-97c6-48f82e7c7bdc" containerName="maas-api" Nov 04 01:43:05.053445 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:05.053425 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzrrd" Nov 04 01:43:05.059957 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:05.059905 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzrrd"] Nov 04 01:43:05.113999 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:05.113972 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:43:05.114121 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:05.114001 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:43:05.144711 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:05.144685 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05f0273d-e8a3-49c9-91b7-0f4fcf2a564f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qzrrd\" (UID: \"05f0273d-e8a3-49c9-91b7-0f4fcf2a564f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzrrd" Nov 04 01:43:05.144844 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:05.144822 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qsfd\" (UniqueName: \"kubernetes.io/projected/05f0273d-e8a3-49c9-91b7-0f4fcf2a564f-kube-api-access-7qsfd\") pod \"maas-api-revoke-sub-3d4156f1-qzrrd\" (UID: \"05f0273d-e8a3-49c9-91b7-0f4fcf2a564f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzrrd" Nov 04 01:43:05.144899 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:05.144873 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05f0273d-e8a3-49c9-91b7-0f4fcf2a564f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qzrrd\" (UID: \"05f0273d-e8a3-49c9-91b7-0f4fcf2a564f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzrrd" Nov 04 01:43:05.185575 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:05.185544 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f1b9871-4d63-46d1-ba30-859d7b911d14" path="/var/lib/kubelet/pods/1f1b9871-4d63-46d1-ba30-859d7b911d14/volumes" Nov 04 01:43:05.185958 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:05.185941 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8532c13f-1f8b-4e27-97c6-48f82e7c7bdc" path="/var/lib/kubelet/pods/8532c13f-1f8b-4e27-97c6-48f82e7c7bdc/volumes" Nov 04 01:43:05.245692 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:05.245663 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qsfd\" (UniqueName: \"kubernetes.io/projected/05f0273d-e8a3-49c9-91b7-0f4fcf2a564f-kube-api-access-7qsfd\") pod \"maas-api-revoke-sub-3d4156f1-qzrrd\" (UID: \"05f0273d-e8a3-49c9-91b7-0f4fcf2a564f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzrrd" Nov 04 01:43:05.245843 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:05.245700 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05f0273d-e8a3-49c9-91b7-0f4fcf2a564f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qzrrd\" (UID: \"05f0273d-e8a3-49c9-91b7-0f4fcf2a564f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzrrd" Nov 04 01:43:05.245843 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:05.245763 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05f0273d-e8a3-49c9-91b7-0f4fcf2a564f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qzrrd\" (UID: \"05f0273d-e8a3-49c9-91b7-0f4fcf2a564f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzrrd" Nov 04 01:43:05.246487 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:05.246411 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05f0273d-e8a3-49c9-91b7-0f4fcf2a564f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qzrrd\" (UID: \"05f0273d-e8a3-49c9-91b7-0f4fcf2a564f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzrrd" Nov 04 01:43:05.248213 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:05.248192 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05f0273d-e8a3-49c9-91b7-0f4fcf2a564f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qzrrd\" (UID: \"05f0273d-e8a3-49c9-91b7-0f4fcf2a564f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzrrd" Nov 04 01:43:05.254843 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:05.254819 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qsfd\" (UniqueName: \"kubernetes.io/projected/05f0273d-e8a3-49c9-91b7-0f4fcf2a564f-kube-api-access-7qsfd\") pod \"maas-api-revoke-sub-3d4156f1-qzrrd\" (UID: \"05f0273d-e8a3-49c9-91b7-0f4fcf2a564f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzrrd" Nov 04 01:43:05.371954 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:05.371863 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzrrd" Nov 04 01:43:05.651594 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:05.651568 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzrrd"] Nov 04 01:43:05.652527 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:43:05.652501 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f0273d_e8a3_49c9_91b7_0f4fcf2a564f.slice/crio-b569f701e69cdb062eef03663dd7189c77e8ea8fe96ce94abc4c769503c38756 WatchSource:0}: Error finding container b569f701e69cdb062eef03663dd7189c77e8ea8fe96ce94abc4c769503c38756: Status 404 returned error can't find the container with id b569f701e69cdb062eef03663dd7189c77e8ea8fe96ce94abc4c769503c38756 Nov 04 01:43:06.480988 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:43:06.480953 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:43:06.617960 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:06.617904 7059 generic.go:356] "Generic (PLEG): container finished" podID="05f0273d-e8a3-49c9-91b7-0f4fcf2a564f" containerID="0384ba7f3cccf057dc4e174e18c2879855bb1470a685858799278589b013644f" exitCode=0 Nov 04 01:43:06.618100 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:06.617985 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzrrd" event={"ID":"05f0273d-e8a3-49c9-91b7-0f4fcf2a564f","Type":"ContainerDied","Data":"0384ba7f3cccf057dc4e174e18c2879855bb1470a685858799278589b013644f"} Nov 04 01:43:06.618100 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:06.618042 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzrrd" event={"ID":"05f0273d-e8a3-49c9-91b7-0f4fcf2a564f","Type":"ContainerStarted","Data":"b569f701e69cdb062eef03663dd7189c77e8ea8fe96ce94abc4c769503c38756"} Nov 04 01:43:07.764094 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:07.764071 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzrrd" Nov 04 01:43:07.870659 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:07.870622 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05f0273d-e8a3-49c9-91b7-0f4fcf2a564f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05f0273d-e8a3-49c9-91b7-0f4fcf2a564f-maas-api-openshift-ca\") pod \"05f0273d-e8a3-49c9-91b7-0f4fcf2a564f\" (UID: \"05f0273d-e8a3-49c9-91b7-0f4fcf2a564f\") " Nov 04 01:43:07.870828 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:07.870682 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05f0273d-e8a3-49c9-91b7-0f4fcf2a564f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05f0273d-e8a3-49c9-91b7-0f4fcf2a564f-maas-api-xks-ca\") pod \"05f0273d-e8a3-49c9-91b7-0f4fcf2a564f\" (UID: \"05f0273d-e8a3-49c9-91b7-0f4fcf2a564f\") " Nov 04 01:43:07.870828 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:07.870777 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05f0273d-e8a3-49c9-91b7-0f4fcf2a564f-kube-api-access-7qsfd\" (UniqueName: \"kubernetes.io/projected/05f0273d-e8a3-49c9-91b7-0f4fcf2a564f-kube-api-access-7qsfd\") pod \"05f0273d-e8a3-49c9-91b7-0f4fcf2a564f\" (UID: \"05f0273d-e8a3-49c9-91b7-0f4fcf2a564f\") " Nov 04 01:43:07.870976 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:07.870942 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f0273d-e8a3-49c9-91b7-0f4fcf2a564f-maas-api-openshift-ca" pod "05f0273d-e8a3-49c9-91b7-0f4fcf2a564f" (UID: "05f0273d-e8a3-49c9-91b7-0f4fcf2a564f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:43:07.871159 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:07.871143 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05f0273d-e8a3-49c9-91b7-0f4fcf2a564f-maas-api-openshift-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:07.873102 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:07.873073 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05f0273d-e8a3-49c9-91b7-0f4fcf2a564f-maas-api-xks-ca" pod "05f0273d-e8a3-49c9-91b7-0f4fcf2a564f" (UID: "05f0273d-e8a3-49c9-91b7-0f4fcf2a564f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:43:07.873102 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:07.873079 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f0273d-e8a3-49c9-91b7-0f4fcf2a564f-kube-api-access-7qsfd" pod "05f0273d-e8a3-49c9-91b7-0f4fcf2a564f" (UID: "05f0273d-e8a3-49c9-91b7-0f4fcf2a564f"). InnerVolumeSpecName "kube-api-access-7qsfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:43:07.972102 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:07.972081 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qsfd\" (UniqueName: \"kubernetes.io/projected/05f0273d-e8a3-49c9-91b7-0f4fcf2a564f-kube-api-access-7qsfd\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:07.972102 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:07.972104 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05f0273d-e8a3-49c9-91b7-0f4fcf2a564f-maas-api-xks-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:08.631991 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:08.631959 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzrrd" Nov 04 01:43:08.631991 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:08.631968 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzrrd" event={"ID":"05f0273d-e8a3-49c9-91b7-0f4fcf2a564f","Type":"ContainerDied","Data":"b569f701e69cdb062eef03663dd7189c77e8ea8fe96ce94abc4c769503c38756"} Nov 04 01:43:08.632209 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:08.632017 7059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b569f701e69cdb062eef03663dd7189c77e8ea8fe96ce94abc4c769503c38756" Nov 04 01:43:15.022399 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:15.022367 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w2mnf"] Nov 04 01:43:15.025804 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:15.025748 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w2mnf"] Nov 04 01:43:15.177485 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:15.177449 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00341182-d49f-499c-a3eb-2c26d7987a35" path="/var/lib/kubelet/pods/00341182-d49f-499c-a3eb-2c26d7987a35/volumes" Nov 04 01:43:18.953107 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:18.953053 7059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-7rw6b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:43:18.953540 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:18.953141 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" podUID="994ce0d7-9fe2-494e-a915-94fe06da822e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:43:26.036053 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:26.036017 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sd6ps"] Nov 04 01:43:26.038420 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:26.038398 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sd6ps"] Nov 04 01:43:27.176060 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:27.176024 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cee30f6-a675-4775-b767-b78376d9257c" path="/var/lib/kubelet/pods/5cee30f6-a675-4775-b767-b78376d9257c/volumes" Nov 04 01:43:36.479527 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:43:36.479488 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:43:36.483802 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:36.483776 7059 scope.go:122] "RemoveContainer" containerID="dd63e3327c51bbab4f370f4665c31c39bd216173958ce33fd25f8ddf4c25195b" Nov 04 01:43:46.023784 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:46.023696 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-x9p7x"] Nov 04 01:43:46.025836 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:46.025814 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-x9p7x"] Nov 04 01:43:47.176504 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:43:47.176469 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb" path="/var/lib/kubelet/pods/58c5ef4a-83ee-4b36-8b52-bfe84e7a40eb/volumes" Nov 04 01:44:06.479545 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:44:06.479502 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:44:20.021674 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:20.021640 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xwddf"] Nov 04 01:44:20.024352 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:20.024328 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xwddf"] Nov 04 01:44:21.176504 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:21.176469 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6620248b-44a6-40f4-915f-28721500f1f5" path="/var/lib/kubelet/pods/6620248b-44a6-40f4-915f-28721500f1f5/volumes" Nov 04 01:44:34.024021 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:34.023984 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zfmjd"] Nov 04 01:44:34.026016 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:34.025994 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-zfmjd"] Nov 04 01:44:35.175859 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:35.175824 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13704caf-55c8-4c4c-985f-da9755654491" path="/var/lib/kubelet/pods/13704caf-55c8-4c4c-985f-da9755654491/volumes" Nov 04 01:44:36.479456 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:44:36.479421 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:44:36.512163 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:36.512141 7059 scope.go:122] "RemoveContainer" containerID="20ae6504cabb61f6bbc686071c7e11c7f3482113f38d364e62e913a219af144c" Nov 04 01:44:36.521123 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:36.521108 7059 scope.go:122] "RemoveContainer" containerID="5a63f075d63647f98274aa808935adb4b020ab8606837b99d9af9bfd75272d2b" Nov 04 01:44:36.529758 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:36.529737 7059 scope.go:122] "RemoveContainer" containerID="01caf3df7a4de61e753971f360d0363ef02da43b264e84653604fe670165b686" Nov 04 01:44:36.539574 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:36.539553 7059 scope.go:122] "RemoveContainer" containerID="69bb95b913afa05dde926e61a529a4a4d6d5f412313170aabdd4161e53edcaa1" Nov 04 01:44:44.306704 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:44.306669 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wwhm9"] Nov 04 01:44:44.307890 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:44.307864 7059 cpu_manager.go:414] "containerMap: removing container" podUID="05f0273d-e8a3-49c9-91b7-0f4fcf2a564f" containerName="revoke-keys" Nov 04 01:44:44.308092 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:44.308057 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05f0273d-e8a3-49c9-91b7-0f4fcf2a564f" containerName="revoke-keys" Nov 04 01:44:44.308276 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:44.308258 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="05f0273d-e8a3-49c9-91b7-0f4fcf2a564f" containerName="revoke-keys" Nov 04 01:44:44.311839 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:44.311816 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wwhm9" Nov 04 01:44:44.319325 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:44.319303 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:44:44.319433 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:44.319335 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:44:44.323899 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:44.320818 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wwhm9"] Nov 04 01:44:44.338816 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:44.338790 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92932abf-4280-4b07-80f6-5c4c6fea0f72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-wwhm9\" (UID: \"92932abf-4280-4b07-80f6-5c4c6fea0f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wwhm9" Nov 04 01:44:44.338956 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:44.338829 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqgd2\" (UniqueName: \"kubernetes.io/projected/92932abf-4280-4b07-80f6-5c4c6fea0f72-kube-api-access-rqgd2\") pod \"maas-api-revoke-sub-d576e68b-wwhm9\" (UID: \"92932abf-4280-4b07-80f6-5c4c6fea0f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wwhm9" Nov 04 01:44:44.338956 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:44.338856 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92932abf-4280-4b07-80f6-5c4c6fea0f72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-wwhm9\" (UID: \"92932abf-4280-4b07-80f6-5c4c6fea0f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wwhm9" Nov 04 01:44:44.440244 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:44.440208 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92932abf-4280-4b07-80f6-5c4c6fea0f72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-wwhm9\" (UID: \"92932abf-4280-4b07-80f6-5c4c6fea0f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wwhm9" Nov 04 01:44:44.440382 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:44.440262 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqgd2\" (UniqueName: \"kubernetes.io/projected/92932abf-4280-4b07-80f6-5c4c6fea0f72-kube-api-access-rqgd2\") pod \"maas-api-revoke-sub-d576e68b-wwhm9\" (UID: \"92932abf-4280-4b07-80f6-5c4c6fea0f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wwhm9" Nov 04 01:44:44.440382 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:44.440300 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92932abf-4280-4b07-80f6-5c4c6fea0f72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-wwhm9\" (UID: \"92932abf-4280-4b07-80f6-5c4c6fea0f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wwhm9" Nov 04 01:44:44.441178 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:44.441149 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92932abf-4280-4b07-80f6-5c4c6fea0f72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-wwhm9\" (UID: \"92932abf-4280-4b07-80f6-5c4c6fea0f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wwhm9" Nov 04 01:44:44.442807 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:44.442783 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92932abf-4280-4b07-80f6-5c4c6fea0f72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-wwhm9\" (UID: \"92932abf-4280-4b07-80f6-5c4c6fea0f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wwhm9" Nov 04 01:44:44.447805 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:44.447785 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqgd2\" (UniqueName: \"kubernetes.io/projected/92932abf-4280-4b07-80f6-5c4c6fea0f72-kube-api-access-rqgd2\") pod \"maas-api-revoke-sub-d576e68b-wwhm9\" (UID: \"92932abf-4280-4b07-80f6-5c4c6fea0f72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wwhm9" Nov 04 01:44:44.627524 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:44.627455 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wwhm9" Nov 04 01:44:44.905332 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:44.905305 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wwhm9"] Nov 04 01:44:44.907159 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:44:44.907131 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92932abf_4280_4b07_80f6_5c4c6fea0f72.slice/crio-35e44a6079aa43d7b70ce70436be3b125f35afd0bb94327038227812febfd346 WatchSource:0}: Error finding container 35e44a6079aa43d7b70ce70436be3b125f35afd0bb94327038227812febfd346: Status 404 returned error can't find the container with id 35e44a6079aa43d7b70ce70436be3b125f35afd0bb94327038227812febfd346 Nov 04 01:44:45.242517 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:45.242481 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wwhm9" event={"ID":"92932abf-4280-4b07-80f6-5c4c6fea0f72","Type":"ContainerStarted","Data":"ea8dbc242ad7177a58426e8e76e7a38591e2e6290066de8b1d550788866f3a69"} Nov 04 01:44:45.242632 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:45.242544 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wwhm9" event={"ID":"92932abf-4280-4b07-80f6-5c4c6fea0f72","Type":"ContainerStarted","Data":"35e44a6079aa43d7b70ce70436be3b125f35afd0bb94327038227812febfd346"} Nov 04 01:44:45.260681 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:45.260637 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wwhm9" podStartSLOduration=1.260624889 podStartE2EDuration="1.260624889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:44:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:44:45.25858688 +0000 UTC m=+2770.200435784" watchObservedRunningTime="2026-11-04 01:44:45.260624889 +0000 UTC m=+2770.202473841" Nov 04 01:44:46.249602 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:46.249571 7059 generic.go:356] "Generic (PLEG): container finished" podID="92932abf-4280-4b07-80f6-5c4c6fea0f72" containerID="ea8dbc242ad7177a58426e8e76e7a38591e2e6290066de8b1d550788866f3a69" exitCode=0 Nov 04 01:44:46.249602 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:46.249604 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wwhm9" event={"ID":"92932abf-4280-4b07-80f6-5c4c6fea0f72","Type":"ContainerDied","Data":"ea8dbc242ad7177a58426e8e76e7a38591e2e6290066de8b1d550788866f3a69"} Nov 04 01:44:47.396381 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:47.396356 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wwhm9" Nov 04 01:44:47.468762 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:47.468732 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92932abf-4280-4b07-80f6-5c4c6fea0f72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92932abf-4280-4b07-80f6-5c4c6fea0f72-maas-api-xks-ca\") pod \"92932abf-4280-4b07-80f6-5c4c6fea0f72\" (UID: \"92932abf-4280-4b07-80f6-5c4c6fea0f72\") " Nov 04 01:44:47.468967 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:47.468854 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92932abf-4280-4b07-80f6-5c4c6fea0f72-kube-api-access-rqgd2\" (UniqueName: \"kubernetes.io/projected/92932abf-4280-4b07-80f6-5c4c6fea0f72-kube-api-access-rqgd2\") pod \"92932abf-4280-4b07-80f6-5c4c6fea0f72\" (UID: \"92932abf-4280-4b07-80f6-5c4c6fea0f72\") " Nov 04 01:44:47.469048 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:47.468975 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92932abf-4280-4b07-80f6-5c4c6fea0f72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92932abf-4280-4b07-80f6-5c4c6fea0f72-maas-api-openshift-ca\") pod \"92932abf-4280-4b07-80f6-5c4c6fea0f72\" (UID: \"92932abf-4280-4b07-80f6-5c4c6fea0f72\") " Nov 04 01:44:47.469331 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:47.469309 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92932abf-4280-4b07-80f6-5c4c6fea0f72-maas-api-openshift-ca" pod "92932abf-4280-4b07-80f6-5c4c6fea0f72" (UID: "92932abf-4280-4b07-80f6-5c4c6fea0f72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:44:47.470988 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:47.470964 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92932abf-4280-4b07-80f6-5c4c6fea0f72-maas-api-xks-ca" pod "92932abf-4280-4b07-80f6-5c4c6fea0f72" (UID: "92932abf-4280-4b07-80f6-5c4c6fea0f72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:44:47.471259 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:47.471237 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92932abf-4280-4b07-80f6-5c4c6fea0f72-kube-api-access-rqgd2" pod "92932abf-4280-4b07-80f6-5c4c6fea0f72" (UID: "92932abf-4280-4b07-80f6-5c4c6fea0f72"). InnerVolumeSpecName "kube-api-access-rqgd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:44:47.570757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:47.570674 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqgd2\" (UniqueName: \"kubernetes.io/projected/92932abf-4280-4b07-80f6-5c4c6fea0f72-kube-api-access-rqgd2\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:47.570757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:47.570712 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92932abf-4280-4b07-80f6-5c4c6fea0f72-maas-api-openshift-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:47.570757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:47.570726 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92932abf-4280-4b07-80f6-5c4c6fea0f72-maas-api-xks-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:48.263469 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:48.263432 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wwhm9" event={"ID":"92932abf-4280-4b07-80f6-5c4c6fea0f72","Type":"ContainerDied","Data":"35e44a6079aa43d7b70ce70436be3b125f35afd0bb94327038227812febfd346"} Nov 04 01:44:48.263469 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:48.263459 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wwhm9" Nov 04 01:44:48.263732 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:44:48.263491 7059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35e44a6079aa43d7b70ce70436be3b125f35afd0bb94327038227812febfd346" Nov 04 01:45:00.131010 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:00.130962 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895945-9bpzr"] Nov 04 01:45:00.131723 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:00.131585 7059 cpu_manager.go:414] "containerMap: removing container" podUID="92932abf-4280-4b07-80f6-5c4c6fea0f72" containerName="revoke-keys" Nov 04 01:45:00.131723 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:00.131606 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92932abf-4280-4b07-80f6-5c4c6fea0f72" containerName="revoke-keys" Nov 04 01:45:00.131723 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:00.131682 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="92932abf-4280-4b07-80f6-5c4c6fea0f72" containerName="revoke-keys" Nov 04 01:45:00.135204 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:00.135182 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-9bpzr" Nov 04 01:45:00.149117 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:00.149090 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:45:00.149752 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:00.149668 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895945-9bpzr"] Nov 04 01:45:00.224544 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:00.224494 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40e28f0e-5319-4995-ba22-5ba2e7ea1a53-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895945-9bpzr\" (UID: \"40e28f0e-5319-4995-ba22-5ba2e7ea1a53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-9bpzr" Nov 04 01:45:00.226200 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:00.226164 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvtcz\" (UniqueName: \"kubernetes.io/projected/40e28f0e-5319-4995-ba22-5ba2e7ea1a53-kube-api-access-wvtcz\") pod \"maas-api-key-cleanup-29895945-9bpzr\" (UID: \"40e28f0e-5319-4995-ba22-5ba2e7ea1a53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-9bpzr" Nov 04 01:45:00.327702 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:00.327671 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40e28f0e-5319-4995-ba22-5ba2e7ea1a53-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895945-9bpzr\" (UID: \"40e28f0e-5319-4995-ba22-5ba2e7ea1a53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-9bpzr" Nov 04 01:45:00.327868 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:00.327717 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvtcz\" (UniqueName: \"kubernetes.io/projected/40e28f0e-5319-4995-ba22-5ba2e7ea1a53-kube-api-access-wvtcz\") pod \"maas-api-key-cleanup-29895945-9bpzr\" (UID: \"40e28f0e-5319-4995-ba22-5ba2e7ea1a53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-9bpzr" Nov 04 01:45:00.328373 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:00.328351 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40e28f0e-5319-4995-ba22-5ba2e7ea1a53-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895945-9bpzr\" (UID: \"40e28f0e-5319-4995-ba22-5ba2e7ea1a53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-9bpzr" Nov 04 01:45:00.337257 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:00.337222 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvtcz\" (UniqueName: \"kubernetes.io/projected/40e28f0e-5319-4995-ba22-5ba2e7ea1a53-kube-api-access-wvtcz\") pod \"maas-api-key-cleanup-29895945-9bpzr\" (UID: \"40e28f0e-5319-4995-ba22-5ba2e7ea1a53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-9bpzr" Nov 04 01:45:00.524751 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:00.524716 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-9bpzr" Nov 04 01:45:00.802123 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:00.802090 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895945-9bpzr"] Nov 04 01:45:01.352811 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:01.352731 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-9bpzr" event={"ID":"40e28f0e-5319-4995-ba22-5ba2e7ea1a53","Type":"ContainerStarted","Data":"8b6f33ec9bc30d4c2e04df0cf95afe1687ab68188ff61b02c084d85522ec7405"} Nov 04 01:45:01.352811 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:01.352786 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-9bpzr" event={"ID":"40e28f0e-5319-4995-ba22-5ba2e7ea1a53","Type":"ContainerStarted","Data":"03633f9c0715d78bc19771ba55f123c2f30801278842748bab94552c1ba02ce6"} Nov 04 01:45:01.372607 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:01.372559 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-9bpzr" podStartSLOduration=1.372545041 podStartE2EDuration="1.372545041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:45:01.37050264 +0000 UTC m=+2786.312351544" watchObservedRunningTime="2026-11-04 01:45:01.372545041 +0000 UTC m=+2786.314393943" Nov 04 01:45:02.360822 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:02.360786 7059 generic.go:356] "Generic (PLEG): container finished" podID="40e28f0e-5319-4995-ba22-5ba2e7ea1a53" containerID="8b6f33ec9bc30d4c2e04df0cf95afe1687ab68188ff61b02c084d85522ec7405" exitCode=60 Nov 04 01:45:02.361266 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:02.360867 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-9bpzr" event={"ID":"40e28f0e-5319-4995-ba22-5ba2e7ea1a53","Type":"ContainerDied","Data":"8b6f33ec9bc30d4c2e04df0cf95afe1687ab68188ff61b02c084d85522ec7405"} Nov 04 01:45:02.361266 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:02.361152 7059 scope.go:122] "RemoveContainer" containerID="8b6f33ec9bc30d4c2e04df0cf95afe1687ab68188ff61b02c084d85522ec7405" Nov 04 01:45:03.368281 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:03.368245 7059 generic.go:356] "Generic (PLEG): container finished" podID="40e28f0e-5319-4995-ba22-5ba2e7ea1a53" containerID="120b8ac2eab5f012f9b47ab1d748db7143ece948614e294501d097adcb79d5f5" exitCode=60 Nov 04 01:45:03.368797 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:03.368332 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-9bpzr" event={"ID":"40e28f0e-5319-4995-ba22-5ba2e7ea1a53","Type":"ContainerDied","Data":"120b8ac2eab5f012f9b47ab1d748db7143ece948614e294501d097adcb79d5f5"} Nov 04 01:45:03.368797 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:03.368393 7059 scope.go:122] "RemoveContainer" containerID="8b6f33ec9bc30d4c2e04df0cf95afe1687ab68188ff61b02c084d85522ec7405" Nov 04 01:45:03.368797 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:03.368666 7059 scope.go:122] "RemoveContainer" containerID="120b8ac2eab5f012f9b47ab1d748db7143ece948614e294501d097adcb79d5f5" Nov 04 01:45:03.368976 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:45:03.368940 7059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895945-9bpzr_odh-ai-gateway-infra(40e28f0e-5319-4995-ba22-5ba2e7ea1a53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-9bpzr" podUID="40e28f0e-5319-4995-ba22-5ba2e7ea1a53" Nov 04 01:45:04.376383 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:04.376356 7059 scope.go:122] "RemoveContainer" containerID="120b8ac2eab5f012f9b47ab1d748db7143ece948614e294501d097adcb79d5f5" Nov 04 01:45:04.376805 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:45:04.376569 7059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895945-9bpzr_odh-ai-gateway-infra(40e28f0e-5319-4995-ba22-5ba2e7ea1a53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-9bpzr" podUID="40e28f0e-5319-4995-ba22-5ba2e7ea1a53" Nov 04 01:45:06.479905 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:45:06.479864 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:45:16.171054 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:16.171023 7059 scope.go:122] "RemoveContainer" containerID="120b8ac2eab5f012f9b47ab1d748db7143ece948614e294501d097adcb79d5f5" Nov 04 01:45:17.456102 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:17.456071 7059 generic.go:356] "Generic (PLEG): container finished" podID="40e28f0e-5319-4995-ba22-5ba2e7ea1a53" containerID="a91d7759100a09821376808a1dcd635ff8664f2e29cc4ddf9a319ec3377f9374" exitCode=60 Nov 04 01:45:17.456525 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:17.456148 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-9bpzr" event={"ID":"40e28f0e-5319-4995-ba22-5ba2e7ea1a53","Type":"ContainerDied","Data":"a91d7759100a09821376808a1dcd635ff8664f2e29cc4ddf9a319ec3377f9374"} Nov 04 01:45:17.456525 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:17.456207 7059 scope.go:122] "RemoveContainer" containerID="120b8ac2eab5f012f9b47ab1d748db7143ece948614e294501d097adcb79d5f5" Nov 04 01:45:17.456645 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:17.456531 7059 scope.go:122] "RemoveContainer" containerID="a91d7759100a09821376808a1dcd635ff8664f2e29cc4ddf9a319ec3377f9374" Nov 04 01:45:17.456813 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:45:17.456790 7059 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895945-9bpzr_odh-ai-gateway-infra(40e28f0e-5319-4995-ba22-5ba2e7ea1a53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-9bpzr" podUID="40e28f0e-5319-4995-ba22-5ba2e7ea1a53" Nov 04 01:45:17.935983 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:17.935946 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v"] Nov 04 01:45:17.942025 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:17.941998 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v" Nov 04 01:45:17.948834 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:17.948799 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v"] Nov 04 01:45:17.952496 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:17.952469 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-2ba03d-serving-cert\"" Nov 04 01:45:17.955214 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:17.954027 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 04 01:45:17.955214 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:17.954085 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nb79d\"" Nov 04 01:45:18.006086 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:18.006048 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5f6de77-e129-4a60-bc12-1b6e8324dd57-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v\" (UID: \"d5f6de77-e129-4a60-bc12-1b6e8324dd57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v" Nov 04 01:45:18.006279 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:18.006124 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5f6de77-e129-4a60-bc12-1b6e8324dd57-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v\" (UID: \"d5f6de77-e129-4a60-bc12-1b6e8324dd57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v" Nov 04 01:45:18.006349 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:18.006280 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5f6de77-e129-4a60-bc12-1b6e8324dd57-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v\" (UID: \"d5f6de77-e129-4a60-bc12-1b6e8324dd57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v" Nov 04 01:45:18.006349 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:18.006327 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq4vl\" (UniqueName: \"kubernetes.io/projected/d5f6de77-e129-4a60-bc12-1b6e8324dd57-kube-api-access-jq4vl\") pod \"maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v\" (UID: \"d5f6de77-e129-4a60-bc12-1b6e8324dd57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v" Nov 04 01:45:18.107057 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:18.107023 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5f6de77-e129-4a60-bc12-1b6e8324dd57-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v\" (UID: \"d5f6de77-e129-4a60-bc12-1b6e8324dd57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v" Nov 04 01:45:18.107245 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:18.107130 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5f6de77-e129-4a60-bc12-1b6e8324dd57-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v\" (UID: \"d5f6de77-e129-4a60-bc12-1b6e8324dd57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v" Nov 04 01:45:18.107245 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:18.107164 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq4vl\" (UniqueName: \"kubernetes.io/projected/d5f6de77-e129-4a60-bc12-1b6e8324dd57-kube-api-access-jq4vl\") pod \"maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v\" (UID: \"d5f6de77-e129-4a60-bc12-1b6e8324dd57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v" Nov 04 01:45:18.107381 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:18.107253 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5f6de77-e129-4a60-bc12-1b6e8324dd57-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v\" (UID: \"d5f6de77-e129-4a60-bc12-1b6e8324dd57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v" Nov 04 01:45:18.108277 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:18.108246 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5f6de77-e129-4a60-bc12-1b6e8324dd57-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v\" (UID: \"d5f6de77-e129-4a60-bc12-1b6e8324dd57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v" Nov 04 01:45:18.109904 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:18.109875 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5f6de77-e129-4a60-bc12-1b6e8324dd57-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v\" (UID: \"d5f6de77-e129-4a60-bc12-1b6e8324dd57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v" Nov 04 01:45:18.110486 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:18.110466 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5f6de77-e129-4a60-bc12-1b6e8324dd57-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v\" (UID: \"d5f6de77-e129-4a60-bc12-1b6e8324dd57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v" Nov 04 01:45:18.119194 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:18.119167 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq4vl\" (UniqueName: \"kubernetes.io/projected/d5f6de77-e129-4a60-bc12-1b6e8324dd57-kube-api-access-jq4vl\") pod \"maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v\" (UID: \"d5f6de77-e129-4a60-bc12-1b6e8324dd57\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v" Nov 04 01:45:18.260761 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:18.260676 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v" Nov 04 01:45:18.484514 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:18.484478 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895945-9bpzr"] Nov 04 01:45:18.542857 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:18.542833 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v"] Nov 04 01:45:18.545504 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:45:18.545471 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5f6de77_e129_4a60_bc12_1b6e8324dd57.slice/crio-ec6a0b0efb0327c86c1f7800622c00bedc0152f65c058aafd886c07afdbcfbe6 WatchSource:0}: Error finding container ec6a0b0efb0327c86c1f7800622c00bedc0152f65c058aafd886c07afdbcfbe6: Status 404 returned error can't find the container with id ec6a0b0efb0327c86c1f7800622c00bedc0152f65c058aafd886c07afdbcfbe6 Nov 04 01:45:18.621791 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:18.621763 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-9bpzr" Nov 04 01:45:18.713512 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:18.713452 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40e28f0e-5319-4995-ba22-5ba2e7ea1a53-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40e28f0e-5319-4995-ba22-5ba2e7ea1a53-maas-api-service-ca\") pod \"40e28f0e-5319-4995-ba22-5ba2e7ea1a53\" (UID: \"40e28f0e-5319-4995-ba22-5ba2e7ea1a53\") " Nov 04 01:45:18.713512 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:18.713504 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40e28f0e-5319-4995-ba22-5ba2e7ea1a53-kube-api-access-wvtcz\" (UniqueName: \"kubernetes.io/projected/40e28f0e-5319-4995-ba22-5ba2e7ea1a53-kube-api-access-wvtcz\") pod \"40e28f0e-5319-4995-ba22-5ba2e7ea1a53\" (UID: \"40e28f0e-5319-4995-ba22-5ba2e7ea1a53\") " Nov 04 01:45:18.713839 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:18.713780 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e28f0e-5319-4995-ba22-5ba2e7ea1a53-maas-api-service-ca" pod "40e28f0e-5319-4995-ba22-5ba2e7ea1a53" (UID: "40e28f0e-5319-4995-ba22-5ba2e7ea1a53"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:45:18.715596 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:18.715568 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e28f0e-5319-4995-ba22-5ba2e7ea1a53-kube-api-access-wvtcz" pod "40e28f0e-5319-4995-ba22-5ba2e7ea1a53" (UID: "40e28f0e-5319-4995-ba22-5ba2e7ea1a53"). InnerVolumeSpecName "kube-api-access-wvtcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:45:18.815192 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:18.815111 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40e28f0e-5319-4995-ba22-5ba2e7ea1a53-maas-api-service-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:45:18.815192 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:18.815143 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvtcz\" (UniqueName: \"kubernetes.io/projected/40e28f0e-5319-4995-ba22-5ba2e7ea1a53-kube-api-access-wvtcz\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:45:19.472836 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:19.472800 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v" event={"ID":"d5f6de77-e129-4a60-bc12-1b6e8324dd57","Type":"ContainerStarted","Data":"c1f0d2e90e68e6f454ad641d53eaa11243045d8d4aa37530944f5f5a4c8b5176"} Nov 04 01:45:19.473065 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:19.472854 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v" event={"ID":"d5f6de77-e129-4a60-bc12-1b6e8324dd57","Type":"ContainerStarted","Data":"ec6a0b0efb0327c86c1f7800622c00bedc0152f65c058aafd886c07afdbcfbe6"} Nov 04 01:45:19.473065 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:19.472918 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v" Nov 04 01:45:19.474336 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:19.474317 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-9bpzr" Nov 04 01:45:19.474481 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:19.474358 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-9bpzr" event={"ID":"40e28f0e-5319-4995-ba22-5ba2e7ea1a53","Type":"ContainerDied","Data":"03633f9c0715d78bc19771ba55f123c2f30801278842748bab94552c1ba02ce6"} Nov 04 01:45:19.474481 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:19.474413 7059 scope.go:122] "RemoveContainer" containerID="a91d7759100a09821376808a1dcd635ff8664f2e29cc4ddf9a319ec3377f9374" Nov 04 01:45:19.500100 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:19.500058 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v" podStartSLOduration=1.9767646399999999 podStartE2EDuration="2.500044931s" totalImagesPullingTime="523.280291ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:45:19.496638454 +0000 UTC m=+2804.438487356" watchObservedRunningTime="2026-11-04 01:45:19.500044931 +0000 UTC m=+2804.441893833" Nov 04 01:45:19.520721 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:19.520697 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895945-9bpzr"] Nov 04 01:45:19.527513 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:19.527492 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895945-9bpzr"] Nov 04 01:45:21.175945 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:21.175889 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40e28f0e-5319-4995-ba22-5ba2e7ea1a53" path="/var/lib/kubelet/pods/40e28f0e-5319-4995-ba22-5ba2e7ea1a53/volumes" Nov 04 01:45:25.487688 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:25.487660 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v" Nov 04 01:45:26.031466 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:26.031425 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n9tzj"] Nov 04 01:45:26.033505 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:26.033482 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-n9tzj"] Nov 04 01:45:27.176869 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:27.176833 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a54b864e-1ded-486d-b8b1-38d87c61dd53" path="/var/lib/kubelet/pods/a54b864e-1ded-486d-b8b1-38d87c61dd53/volumes" Nov 04 01:45:34.074426 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.074386 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm"] Nov 04 01:45:34.075177 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.075124 7059 cpu_manager.go:414] "containerMap: removing container" podUID="40e28f0e-5319-4995-ba22-5ba2e7ea1a53" containerName="cleanup" Nov 04 01:45:34.075177 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.075146 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40e28f0e-5319-4995-ba22-5ba2e7ea1a53" containerName="cleanup" Nov 04 01:45:34.075177 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.075166 7059 cpu_manager.go:414] "containerMap: removing container" podUID="40e28f0e-5319-4995-ba22-5ba2e7ea1a53" containerName="cleanup" Nov 04 01:45:34.075177 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.075174 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40e28f0e-5319-4995-ba22-5ba2e7ea1a53" containerName="cleanup" Nov 04 01:45:34.075497 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.075220 7059 cpu_manager.go:414] "containerMap: removing container" podUID="40e28f0e-5319-4995-ba22-5ba2e7ea1a53" containerName="cleanup" Nov 04 01:45:34.075497 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.075228 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40e28f0e-5319-4995-ba22-5ba2e7ea1a53" containerName="cleanup" Nov 04 01:45:34.075497 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.075330 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="40e28f0e-5319-4995-ba22-5ba2e7ea1a53" containerName="cleanup" Nov 04 01:45:34.075497 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.075348 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="40e28f0e-5319-4995-ba22-5ba2e7ea1a53" containerName="cleanup" Nov 04 01:45:34.079124 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.079104 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm" Nov 04 01:45:34.088529 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.086917 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-2ba03d\"/\"openshift-service-ca.crt\"" Nov 04 01:45:34.088529 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.086918 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-2ba03d\"/\"default-dockercfg-nhdrz\"" Nov 04 01:45:34.088529 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.087024 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-2ba03d\"/\"simulator-single-main-2ba03d-kserve-self-signed-certs\"" Nov 04 01:45:34.088529 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.087036 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-2ba03d\"/\"kube-root-ca.crt\"" Nov 04 01:45:34.088529 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.087110 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm"] Nov 04 01:45:34.169277 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.169242 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-dshm\") pod \"simulator-single-main-2ba03d-kserve-77df94484-zvhkm\" (UID: \"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd\") " pod="e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm" Nov 04 01:45:34.169439 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.169296 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-model-cache\") pod \"simulator-single-main-2ba03d-kserve-77df94484-zvhkm\" (UID: \"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd\") " pod="e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm" Nov 04 01:45:34.169439 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.169396 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcgws\" (UniqueName: \"kubernetes.io/projected/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-kube-api-access-qcgws\") pod \"simulator-single-main-2ba03d-kserve-77df94484-zvhkm\" (UID: \"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd\") " pod="e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm" Nov 04 01:45:34.169520 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.169498 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-tmp-dir\") pod \"simulator-single-main-2ba03d-kserve-77df94484-zvhkm\" (UID: \"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd\") " pod="e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm" Nov 04 01:45:34.169557 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.169548 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-tls-certs\") pod \"simulator-single-main-2ba03d-kserve-77df94484-zvhkm\" (UID: \"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd\") " pod="e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm" Nov 04 01:45:34.169619 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.169607 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-home\") pod \"simulator-single-main-2ba03d-kserve-77df94484-zvhkm\" (UID: \"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd\") " pod="e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm" Nov 04 01:45:34.271028 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.270985 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcgws\" (UniqueName: \"kubernetes.io/projected/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-kube-api-access-qcgws\") pod \"simulator-single-main-2ba03d-kserve-77df94484-zvhkm\" (UID: \"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd\") " pod="e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm" Nov 04 01:45:34.271240 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.271086 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-tmp-dir\") pod \"simulator-single-main-2ba03d-kserve-77df94484-zvhkm\" (UID: \"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd\") " pod="e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm" Nov 04 01:45:34.271240 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.271144 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-tls-certs\") pod \"simulator-single-main-2ba03d-kserve-77df94484-zvhkm\" (UID: \"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd\") " pod="e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm" Nov 04 01:45:34.271240 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.271201 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-home\") pod \"simulator-single-main-2ba03d-kserve-77df94484-zvhkm\" (UID: \"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd\") " pod="e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm" Nov 04 01:45:34.271240 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.271236 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-dshm\") pod \"simulator-single-main-2ba03d-kserve-77df94484-zvhkm\" (UID: \"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd\") " pod="e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm" Nov 04 01:45:34.271427 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.271271 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-model-cache\") pod \"simulator-single-main-2ba03d-kserve-77df94484-zvhkm\" (UID: \"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd\") " pod="e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm" Nov 04 01:45:34.271649 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.271628 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-model-cache\") pod \"simulator-single-main-2ba03d-kserve-77df94484-zvhkm\" (UID: \"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd\") " pod="e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm" Nov 04 01:45:34.271829 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.271811 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-home\") pod \"simulator-single-main-2ba03d-kserve-77df94484-zvhkm\" (UID: \"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd\") " pod="e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm" Nov 04 01:45:34.272373 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.272347 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-tmp-dir\") pod \"simulator-single-main-2ba03d-kserve-77df94484-zvhkm\" (UID: \"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd\") " pod="e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm" Nov 04 01:45:34.274152 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.274129 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-dshm\") pod \"simulator-single-main-2ba03d-kserve-77df94484-zvhkm\" (UID: \"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd\") " pod="e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm" Nov 04 01:45:34.274440 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.274415 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-tls-certs\") pod \"simulator-single-main-2ba03d-kserve-77df94484-zvhkm\" (UID: \"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd\") " pod="e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm" Nov 04 01:45:34.280333 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.280311 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcgws\" (UniqueName: \"kubernetes.io/projected/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-kube-api-access-qcgws\") pod \"simulator-single-main-2ba03d-kserve-77df94484-zvhkm\" (UID: \"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd\") " pod="e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm" Nov 04 01:45:34.394850 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.394793 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm" Nov 04 01:45:34.673726 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:34.673696 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm"] Nov 04 01:45:34.675387 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:45:34.675361 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6431f56f_c17a_411d_a2ec_5edcf0b0a0bd.slice/crio-e48d1142faa8fb4659e11fa9f806c7310e784e419eb3339ef394c0d3d97339fc WatchSource:0}: Error finding container e48d1142faa8fb4659e11fa9f806c7310e784e419eb3339ef394c0d3d97339fc: Status 404 returned error can't find the container with id e48d1142faa8fb4659e11fa9f806c7310e784e419eb3339ef394c0d3d97339fc Nov 04 01:45:35.572598 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:35.572561 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm" event={"ID":"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd","Type":"ContainerStarted","Data":"c5856eeb9a3bbfe563f6269a66ab79713f90d6c0b8f879a2115c817d092cc6e9"} Nov 04 01:45:35.573121 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:35.572609 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm" event={"ID":"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd","Type":"ContainerStarted","Data":"e48d1142faa8fb4659e11fa9f806c7310e784e419eb3339ef394c0d3d97339fc"} Nov 04 01:45:35.592416 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:35.592373 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm" podStartSLOduration=1.5923608329999999 podStartE2EDuration="1.592360833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:45:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:45:35.590654017 +0000 UTC m=+2820.532502914" watchObservedRunningTime="2026-11-04 01:45:35.592360833 +0000 UTC m=+2820.534209736" Nov 04 01:45:36.024278 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:36.024245 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xsc4l"] Nov 04 01:45:36.027010 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:36.026986 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xsc4l"] Nov 04 01:45:36.395610 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:36.395535 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm" Nov 04 01:45:36.480840 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:45:36.480805 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:45:36.577364 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:36.577328 7059 scope.go:122] "RemoveContainer" containerID="23c0a0ff7a1f9e1a021752b72b9bd93061973746c395b357d34b271ee959c987" Nov 04 01:45:36.586832 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:36.586806 7059 scope.go:122] "RemoveContainer" containerID="4a81935cd70a9cf69eeee51c5bac1d5b280b74b61b9f2469e82ab1187817da6e" Nov 04 01:45:36.595560 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:36.595536 7059 scope.go:122] "RemoveContainer" containerID="8246577fc36aac14284e4e6c99742f4edf621569fdd940bd0a9804e1503334a4" Nov 04 01:45:37.176778 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:37.176747 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30e20314-9fdb-47a1-bb32-516965d4e310" path="/var/lib/kubelet/pods/30e20314-9fdb-47a1-bb32-516965d4e310/volumes" Nov 04 01:45:44.395736 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:44.395697 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm" Nov 04 01:45:44.409401 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:44.409378 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm" Nov 04 01:45:44.639515 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:45:44.639478 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm" Nov 04 01:46:06.481500 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:46:06.481456 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:46:19.734279 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:19.734243 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm"] Nov 04 01:46:21.603852 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:21.603810 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v"] Nov 04 01:46:21.604380 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:21.604107 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v" podUID="d5f6de77-e129-4a60-bc12-1b6e8324dd57" containerName="maas-api" containerID="cri-o://c1f0d2e90e68e6f454ad641d53eaa11243045d8d4aa37530944f5f5a4c8b5176" gracePeriod=30 Nov 04 01:46:21.865195 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:21.865161 7059 generic.go:356] "Generic (PLEG): container finished" podID="d5f6de77-e129-4a60-bc12-1b6e8324dd57" containerID="c1f0d2e90e68e6f454ad641d53eaa11243045d8d4aa37530944f5f5a4c8b5176" exitCode=0 Nov 04 01:46:21.865357 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:21.865242 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v" event={"ID":"d5f6de77-e129-4a60-bc12-1b6e8324dd57","Type":"ContainerDied","Data":"c1f0d2e90e68e6f454ad641d53eaa11243045d8d4aa37530944f5f5a4c8b5176"} Nov 04 01:46:21.865357 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:21.865300 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v" event={"ID":"d5f6de77-e129-4a60-bc12-1b6e8324dd57","Type":"ContainerDied","Data":"ec6a0b0efb0327c86c1f7800622c00bedc0152f65c058aafd886c07afdbcfbe6"} Nov 04 01:46:21.865357 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:21.865317 7059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec6a0b0efb0327c86c1f7800622c00bedc0152f65c058aafd886c07afdbcfbe6" Nov 04 01:46:21.880002 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:21.879970 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v" Nov 04 01:46:22.064854 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:22.064827 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5f6de77-e129-4a60-bc12-1b6e8324dd57-kube-api-access-jq4vl\" (UniqueName: \"kubernetes.io/projected/d5f6de77-e129-4a60-bc12-1b6e8324dd57-kube-api-access-jq4vl\") pod \"d5f6de77-e129-4a60-bc12-1b6e8324dd57\" (UID: \"d5f6de77-e129-4a60-bc12-1b6e8324dd57\") " Nov 04 01:46:22.065053 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:22.064989 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5f6de77-e129-4a60-bc12-1b6e8324dd57-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5f6de77-e129-4a60-bc12-1b6e8324dd57-openshift-service-ca\") pod \"d5f6de77-e129-4a60-bc12-1b6e8324dd57\" (UID: \"d5f6de77-e129-4a60-bc12-1b6e8324dd57\") " Nov 04 01:46:22.065129 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:22.065105 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5f6de77-e129-4a60-bc12-1b6e8324dd57-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5f6de77-e129-4a60-bc12-1b6e8324dd57-metrics-certs\") pod \"d5f6de77-e129-4a60-bc12-1b6e8324dd57\" (UID: \"d5f6de77-e129-4a60-bc12-1b6e8324dd57\") " Nov 04 01:46:22.065199 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:22.065137 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5f6de77-e129-4a60-bc12-1b6e8324dd57-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5f6de77-e129-4a60-bc12-1b6e8324dd57-maas-api-tls\") pod \"d5f6de77-e129-4a60-bc12-1b6e8324dd57\" (UID: \"d5f6de77-e129-4a60-bc12-1b6e8324dd57\") " Nov 04 01:46:22.065379 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:22.065350 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5f6de77-e129-4a60-bc12-1b6e8324dd57-openshift-service-ca" pod "d5f6de77-e129-4a60-bc12-1b6e8324dd57" (UID: "d5f6de77-e129-4a60-bc12-1b6e8324dd57"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:46:22.065494 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:22.065474 7059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5f6de77-e129-4a60-bc12-1b6e8324dd57-openshift-service-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:22.067176 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:22.067150 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5f6de77-e129-4a60-bc12-1b6e8324dd57-kube-api-access-jq4vl" pod "d5f6de77-e129-4a60-bc12-1b6e8324dd57" (UID: "d5f6de77-e129-4a60-bc12-1b6e8324dd57"). InnerVolumeSpecName "kube-api-access-jq4vl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:46:22.067312 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:22.067184 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5f6de77-e129-4a60-bc12-1b6e8324dd57-metrics-certs" pod "d5f6de77-e129-4a60-bc12-1b6e8324dd57" (UID: "d5f6de77-e129-4a60-bc12-1b6e8324dd57"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:46:22.067312 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:22.067209 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5f6de77-e129-4a60-bc12-1b6e8324dd57-maas-api-tls" pod "d5f6de77-e129-4a60-bc12-1b6e8324dd57" (UID: "d5f6de77-e129-4a60-bc12-1b6e8324dd57"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:46:22.166421 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:22.166362 7059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5f6de77-e129-4a60-bc12-1b6e8324dd57-metrics-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:22.166421 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:22.166385 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5f6de77-e129-4a60-bc12-1b6e8324dd57-maas-api-tls\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:22.166421 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:22.166397 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq4vl\" (UniqueName: \"kubernetes.io/projected/d5f6de77-e129-4a60-bc12-1b6e8324dd57-kube-api-access-jq4vl\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:22.871109 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:22.871081 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v" Nov 04 01:46:22.894251 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:22.894222 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v"] Nov 04 01:46:22.897073 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:22.897042 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-r792v"] Nov 04 01:46:23.176539 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:23.176509 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5f6de77-e129-4a60-bc12-1b6e8324dd57" path="/var/lib/kubelet/pods/d5f6de77-e129-4a60-bc12-1b6e8324dd57/volumes" Nov 04 01:46:34.755623 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:34.755581 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm" podUID="6431f56f-c17a-411d-a2ec-5edcf0b0a0bd" containerName="main" containerID="cri-o://c5856eeb9a3bbfe563f6269a66ab79713f90d6c0b8f879a2115c817d092cc6e9" gracePeriod=45 Nov 04 01:46:34.945549 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:34.945516 7059 generic.go:356] "Generic (PLEG): container finished" podID="6431f56f-c17a-411d-a2ec-5edcf0b0a0bd" containerID="c5856eeb9a3bbfe563f6269a66ab79713f90d6c0b8f879a2115c817d092cc6e9" exitCode=0 Nov 04 01:46:34.945701 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:34.945558 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm" event={"ID":"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd","Type":"ContainerDied","Data":"c5856eeb9a3bbfe563f6269a66ab79713f90d6c0b8f879a2115c817d092cc6e9"} Nov 04 01:46:35.009317 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:35.009258 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm" Nov 04 01:46:35.088666 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:35.088620 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-tls-certs\" (UniqueName: \"kubernetes.io/secret/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-tls-certs\") pod \"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd\" (UID: \"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd\") " Nov 04 01:46:35.088828 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:35.088720 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-dshm\") pod \"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd\" (UID: \"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd\") " Nov 04 01:46:35.088828 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:35.088811 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-home\" (UniqueName: \"kubernetes.io/empty-dir/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-home\") pod \"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd\" (UID: \"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd\") " Nov 04 01:46:35.088915 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:35.088898 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-kube-api-access-qcgws\" (UniqueName: \"kubernetes.io/projected/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-kube-api-access-qcgws\") pod \"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd\" (UID: \"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd\") " Nov 04 01:46:35.089128 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:35.088973 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-model-cache\") pod \"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd\" (UID: \"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd\") " Nov 04 01:46:35.089128 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:35.089065 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-home" pod "6431f56f-c17a-411d-a2ec-5edcf0b0a0bd" (UID: "6431f56f-c17a-411d-a2ec-5edcf0b0a0bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:46:35.089128 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:35.089077 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-tmp-dir\") pod \"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd\" (UID: \"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd\") " Nov 04 01:46:35.089360 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:35.089329 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-tmp-dir" pod "6431f56f-c17a-411d-a2ec-5edcf0b0a0bd" (UID: "6431f56f-c17a-411d-a2ec-5edcf0b0a0bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:46:35.089578 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:35.089468 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-model-cache" pod "6431f56f-c17a-411d-a2ec-5edcf0b0a0bd" (UID: "6431f56f-c17a-411d-a2ec-5edcf0b0a0bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:46:35.089739 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:35.089715 7059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-model-cache\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:35.089815 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:35.089744 7059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-tmp-dir\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:35.089815 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:35.089759 7059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-home\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:35.091077 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:35.091046 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-tls-certs" pod "6431f56f-c17a-411d-a2ec-5edcf0b0a0bd" (UID: "6431f56f-c17a-411d-a2ec-5edcf0b0a0bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:46:35.091160 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:35.091091 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-dshm" pod "6431f56f-c17a-411d-a2ec-5edcf0b0a0bd" (UID: "6431f56f-c17a-411d-a2ec-5edcf0b0a0bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:46:35.091251 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:35.091232 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-kube-api-access-qcgws" pod "6431f56f-c17a-411d-a2ec-5edcf0b0a0bd" (UID: "6431f56f-c17a-411d-a2ec-5edcf0b0a0bd"). InnerVolumeSpecName "kube-api-access-qcgws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:46:35.190995 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:35.190969 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcgws\" (UniqueName: \"kubernetes.io/projected/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-kube-api-access-qcgws\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:35.190995 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:35.190994 7059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-tls-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:35.191144 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:35.191003 7059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd-dshm\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:35.953254 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:35.953218 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm" event={"ID":"6431f56f-c17a-411d-a2ec-5edcf0b0a0bd","Type":"ContainerDied","Data":"e48d1142faa8fb4659e11fa9f806c7310e784e419eb3339ef394c0d3d97339fc"} Nov 04 01:46:35.953254 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:35.953252 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm" Nov 04 01:46:35.953735 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:35.953272 7059 scope.go:122] "RemoveContainer" containerID="c5856eeb9a3bbfe563f6269a66ab79713f90d6c0b8f879a2115c817d092cc6e9" Nov 04 01:46:35.971721 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:35.971695 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm"] Nov 04 01:46:35.974131 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:35.974111 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-2ba03d/simulator-single-main-2ba03d-kserve-77df94484-zvhkm"] Nov 04 01:46:36.480350 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:46:36.480301 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:46:36.634505 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:36.634478 7059 scope.go:122] "RemoveContainer" containerID="ef5c88ff7b204a76eca53d37a74c8aed2888c81570a3f62cd23b3586e8ba05ce" Nov 04 01:46:36.643489 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:36.643469 7059 scope.go:122] "RemoveContainer" containerID="eb94f69400fb883910cdf4fbae9d847260e35012b11d77070bac9bccfe6ff84f" Nov 04 01:46:36.651588 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:36.651565 7059 scope.go:122] "RemoveContainer" containerID="220ffb6d915e850f755f09745cc8fe08cf1d393b287def7740c8a47db008d99d" Nov 04 01:46:36.659358 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:36.659343 7059 scope.go:122] "RemoveContainer" containerID="c6c0cecac85e2ef57309a6d66f8e5aa9c8bfc426a413843b94dbc6a0b187c18e" Nov 04 01:46:37.176620 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:37.176588 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6431f56f-c17a-411d-a2ec-5edcf0b0a0bd" path="/var/lib/kubelet/pods/6431f56f-c17a-411d-a2ec-5edcf0b0a0bd/volumes" Nov 04 01:46:53.065486 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:53.065404 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp"] Nov 04 01:46:53.066155 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:53.066134 7059 cpu_manager.go:414] "containerMap: removing container" podUID="d5f6de77-e129-4a60-bc12-1b6e8324dd57" containerName="maas-api" Nov 04 01:46:53.066256 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:53.066158 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5f6de77-e129-4a60-bc12-1b6e8324dd57" containerName="maas-api" Nov 04 01:46:53.066256 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:53.066220 7059 cpu_manager.go:414] "containerMap: removing container" podUID="6431f56f-c17a-411d-a2ec-5edcf0b0a0bd" containerName="main" Nov 04 01:46:53.066256 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:53.066230 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6431f56f-c17a-411d-a2ec-5edcf0b0a0bd" containerName="main" Nov 04 01:46:53.066410 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:53.066344 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="6431f56f-c17a-411d-a2ec-5edcf0b0a0bd" containerName="main" Nov 04 01:46:53.066410 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:53.066360 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5f6de77-e129-4a60-bc12-1b6e8324dd57" containerName="maas-api" Nov 04 01:46:53.066410 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:53.066373 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="40e28f0e-5319-4995-ba22-5ba2e7ea1a53" containerName="cleanup" Nov 04 01:46:53.071626 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:53.071604 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp" Nov 04 01:46:53.081534 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:53.081475 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-2ba03d-serving-cert\"" Nov 04 01:46:53.082248 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:53.082224 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nb79d\"" Nov 04 01:46:53.082406 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:53.082298 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 04 01:46:53.083516 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:53.083493 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp"] Nov 04 01:46:53.159526 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:53.159490 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18a81e6-1b83-42f0-b803-6cab76be0bcc-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp\" (UID: \"e18a81e6-1b83-42f0-b803-6cab76be0bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp" Nov 04 01:46:53.159526 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:53.159530 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d568l\" (UniqueName: \"kubernetes.io/projected/e18a81e6-1b83-42f0-b803-6cab76be0bcc-kube-api-access-d568l\") pod \"maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp\" (UID: \"e18a81e6-1b83-42f0-b803-6cab76be0bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp" Nov 04 01:46:53.159791 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:53.159676 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e18a81e6-1b83-42f0-b803-6cab76be0bcc-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp\" (UID: \"e18a81e6-1b83-42f0-b803-6cab76be0bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp" Nov 04 01:46:53.159791 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:53.159745 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18a81e6-1b83-42f0-b803-6cab76be0bcc-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp\" (UID: \"e18a81e6-1b83-42f0-b803-6cab76be0bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp" Nov 04 01:46:53.260721 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:53.260692 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18a81e6-1b83-42f0-b803-6cab76be0bcc-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp\" (UID: \"e18a81e6-1b83-42f0-b803-6cab76be0bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp" Nov 04 01:46:53.260904 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:53.260726 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d568l\" (UniqueName: \"kubernetes.io/projected/e18a81e6-1b83-42f0-b803-6cab76be0bcc-kube-api-access-d568l\") pod \"maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp\" (UID: \"e18a81e6-1b83-42f0-b803-6cab76be0bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp" Nov 04 01:46:53.260904 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:53.260769 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e18a81e6-1b83-42f0-b803-6cab76be0bcc-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp\" (UID: \"e18a81e6-1b83-42f0-b803-6cab76be0bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp" Nov 04 01:46:53.260904 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:53.260800 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18a81e6-1b83-42f0-b803-6cab76be0bcc-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp\" (UID: \"e18a81e6-1b83-42f0-b803-6cab76be0bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp" Nov 04 01:46:53.261569 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:53.261538 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18a81e6-1b83-42f0-b803-6cab76be0bcc-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp\" (UID: \"e18a81e6-1b83-42f0-b803-6cab76be0bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp" Nov 04 01:46:53.263288 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:53.263260 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18a81e6-1b83-42f0-b803-6cab76be0bcc-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp\" (UID: \"e18a81e6-1b83-42f0-b803-6cab76be0bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp" Nov 04 01:46:53.263378 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:53.263362 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e18a81e6-1b83-42f0-b803-6cab76be0bcc-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp\" (UID: \"e18a81e6-1b83-42f0-b803-6cab76be0bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp" Nov 04 01:46:53.274148 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:53.274118 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d568l\" (UniqueName: \"kubernetes.io/projected/e18a81e6-1b83-42f0-b803-6cab76be0bcc-kube-api-access-d568l\") pod \"maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp\" (UID: \"e18a81e6-1b83-42f0-b803-6cab76be0bcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp" Nov 04 01:46:53.388202 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:53.388114 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp" Nov 04 01:46:53.668953 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:53.668909 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp"] Nov 04 01:46:53.670884 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:46:53.670850 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode18a81e6_1b83_42f0_b803_6cab76be0bcc.slice/crio-c0819e4d7259db84fb9711b23af0309576b0e689a90a8235c546b957e7d2a0d8 WatchSource:0}: Error finding container c0819e4d7259db84fb9711b23af0309576b0e689a90a8235c546b957e7d2a0d8: Status 404 returned error can't find the container with id c0819e4d7259db84fb9711b23af0309576b0e689a90a8235c546b957e7d2a0d8 Nov 04 01:46:53.672172 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:53.672153 7059 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:46:54.051229 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:54.051189 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp" event={"ID":"e18a81e6-1b83-42f0-b803-6cab76be0bcc","Type":"ContainerStarted","Data":"c0819e4d7259db84fb9711b23af0309576b0e689a90a8235c546b957e7d2a0d8"} Nov 04 01:46:55.058518 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:55.058471 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp" event={"ID":"e18a81e6-1b83-42f0-b803-6cab76be0bcc","Type":"ContainerStarted","Data":"d5d002a5abb18ca1f131f1dea07686ce77e639dbc0e8acb86b4a2720f30e148a"} Nov 04 01:46:55.059002 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:55.058564 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp" Nov 04 01:46:55.081089 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:46:55.081047 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp" podStartSLOduration=1.411985276 podStartE2EDuration="2.081035462s" totalImagesPullingTime="669.050186ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:46:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:46:55.079895095 +0000 UTC m=+2900.021743998" watchObservedRunningTime="2026-11-04 01:46:55.081035462 +0000 UTC m=+2900.022884364" Nov 04 01:47:01.071502 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:01.071471 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp" Nov 04 01:47:06.480324 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:47:06.480281 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:47:12.381587 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:12.381508 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp"] Nov 04 01:47:12.382138 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:12.381755 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp" podUID="e18a81e6-1b83-42f0-b803-6cab76be0bcc" containerName="maas-api" containerID="cri-o://d5d002a5abb18ca1f131f1dea07686ce77e639dbc0e8acb86b4a2720f30e148a" gracePeriod=30 Nov 04 01:47:12.394512 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:47:12.394477 7059 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode18a81e6_1b83_42f0_b803_6cab76be0bcc.slice/crio-d5d002a5abb18ca1f131f1dea07686ce77e639dbc0e8acb86b4a2720f30e148a.scope\": RecentStats: unable to find data in memory cache]" Nov 04 01:47:12.655187 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:12.655162 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp" Nov 04 01:47:12.764136 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:12.764104 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e18a81e6-1b83-42f0-b803-6cab76be0bcc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18a81e6-1b83-42f0-b803-6cab76be0bcc-metrics-certs\") pod \"e18a81e6-1b83-42f0-b803-6cab76be0bcc\" (UID: \"e18a81e6-1b83-42f0-b803-6cab76be0bcc\") " Nov 04 01:47:12.764319 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:12.764220 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e18a81e6-1b83-42f0-b803-6cab76be0bcc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e18a81e6-1b83-42f0-b803-6cab76be0bcc-maas-api-tls\") pod \"e18a81e6-1b83-42f0-b803-6cab76be0bcc\" (UID: \"e18a81e6-1b83-42f0-b803-6cab76be0bcc\") " Nov 04 01:47:12.764394 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:12.764376 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e18a81e6-1b83-42f0-b803-6cab76be0bcc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18a81e6-1b83-42f0-b803-6cab76be0bcc-openshift-service-ca\") pod \"e18a81e6-1b83-42f0-b803-6cab76be0bcc\" (UID: \"e18a81e6-1b83-42f0-b803-6cab76be0bcc\") " Nov 04 01:47:12.764452 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:12.764413 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e18a81e6-1b83-42f0-b803-6cab76be0bcc-kube-api-access-d568l\" (UniqueName: \"kubernetes.io/projected/e18a81e6-1b83-42f0-b803-6cab76be0bcc-kube-api-access-d568l\") pod \"e18a81e6-1b83-42f0-b803-6cab76be0bcc\" (UID: \"e18a81e6-1b83-42f0-b803-6cab76be0bcc\") " Nov 04 01:47:12.764686 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:12.764661 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e18a81e6-1b83-42f0-b803-6cab76be0bcc-openshift-service-ca" pod "e18a81e6-1b83-42f0-b803-6cab76be0bcc" (UID: "e18a81e6-1b83-42f0-b803-6cab76be0bcc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:47:12.766408 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:12.766378 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e18a81e6-1b83-42f0-b803-6cab76be0bcc-maas-api-tls" pod "e18a81e6-1b83-42f0-b803-6cab76be0bcc" (UID: "e18a81e6-1b83-42f0-b803-6cab76be0bcc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:47:12.766538 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:12.766518 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e18a81e6-1b83-42f0-b803-6cab76be0bcc-metrics-certs" pod "e18a81e6-1b83-42f0-b803-6cab76be0bcc" (UID: "e18a81e6-1b83-42f0-b803-6cab76be0bcc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:47:12.766583 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:12.766564 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e18a81e6-1b83-42f0-b803-6cab76be0bcc-kube-api-access-d568l" pod "e18a81e6-1b83-42f0-b803-6cab76be0bcc" (UID: "e18a81e6-1b83-42f0-b803-6cab76be0bcc"). InnerVolumeSpecName "kube-api-access-d568l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:47:12.865332 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:12.865307 7059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18a81e6-1b83-42f0-b803-6cab76be0bcc-openshift-service-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:12.865332 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:12.865331 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d568l\" (UniqueName: \"kubernetes.io/projected/e18a81e6-1b83-42f0-b803-6cab76be0bcc-kube-api-access-d568l\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:12.865510 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:12.865344 7059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18a81e6-1b83-42f0-b803-6cab76be0bcc-metrics-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:12.865510 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:12.865353 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e18a81e6-1b83-42f0-b803-6cab76be0bcc-maas-api-tls\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:13.161965 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:13.161893 7059 generic.go:356] "Generic (PLEG): container finished" podID="e18a81e6-1b83-42f0-b803-6cab76be0bcc" containerID="d5d002a5abb18ca1f131f1dea07686ce77e639dbc0e8acb86b4a2720f30e148a" exitCode=0 Nov 04 01:47:13.161965 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:13.161968 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp" Nov 04 01:47:13.162181 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:13.161989 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp" event={"ID":"e18a81e6-1b83-42f0-b803-6cab76be0bcc","Type":"ContainerDied","Data":"d5d002a5abb18ca1f131f1dea07686ce77e639dbc0e8acb86b4a2720f30e148a"} Nov 04 01:47:13.162181 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:13.162098 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp" event={"ID":"e18a81e6-1b83-42f0-b803-6cab76be0bcc","Type":"ContainerDied","Data":"c0819e4d7259db84fb9711b23af0309576b0e689a90a8235c546b957e7d2a0d8"} Nov 04 01:47:13.162181 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:13.162129 7059 scope.go:122] "RemoveContainer" containerID="d5d002a5abb18ca1f131f1dea07686ce77e639dbc0e8acb86b4a2720f30e148a" Nov 04 01:47:13.176290 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:13.176274 7059 scope.go:122] "RemoveContainer" containerID="d5d002a5abb18ca1f131f1dea07686ce77e639dbc0e8acb86b4a2720f30e148a" Nov 04 01:47:13.176614 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:47:13.176593 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5d002a5abb18ca1f131f1dea07686ce77e639dbc0e8acb86b4a2720f30e148a\": container with ID starting with d5d002a5abb18ca1f131f1dea07686ce77e639dbc0e8acb86b4a2720f30e148a not found: ID does not exist" containerID="d5d002a5abb18ca1f131f1dea07686ce77e639dbc0e8acb86b4a2720f30e148a" Nov 04 01:47:13.176714 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:13.176622 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5d002a5abb18ca1f131f1dea07686ce77e639dbc0e8acb86b4a2720f30e148a"} err="failed to get container status \"d5d002a5abb18ca1f131f1dea07686ce77e639dbc0e8acb86b4a2720f30e148a\": rpc error: code = NotFound desc = could not find container \"d5d002a5abb18ca1f131f1dea07686ce77e639dbc0e8acb86b4a2720f30e148a\": container with ID starting with d5d002a5abb18ca1f131f1dea07686ce77e639dbc0e8acb86b4a2720f30e148a not found: ID does not exist" Nov 04 01:47:13.187329 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:13.187308 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp"] Nov 04 01:47:13.191315 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:13.191290 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2ba03d-797dbfcc58-2lztp"] Nov 04 01:47:15.177361 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:15.177325 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e18a81e6-1b83-42f0-b803-6cab76be0bcc" path="/var/lib/kubelet/pods/e18a81e6-1b83-42f0-b803-6cab76be0bcc/volumes" Nov 04 01:47:19.421256 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:19.421217 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j"] Nov 04 01:47:19.421951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:19.421903 7059 cpu_manager.go:414] "containerMap: removing container" podUID="e18a81e6-1b83-42f0-b803-6cab76be0bcc" containerName="maas-api" Nov 04 01:47:19.421951 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:19.421939 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e18a81e6-1b83-42f0-b803-6cab76be0bcc" containerName="maas-api" Nov 04 01:47:19.422099 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:19.422035 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e18a81e6-1b83-42f0-b803-6cab76be0bcc" containerName="maas-api" Nov 04 01:47:19.426556 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:19.426540 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j" Nov 04 01:47:19.452891 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:19.452868 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j"] Nov 04 01:47:19.453798 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:19.453773 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-eab16c40-kserve-self-signed-certs\"" Nov 04 01:47:19.532109 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:19.532078 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08e6f78f-275f-4778-b3b4-775fa9f0ac84-tls-certs\") pod \"e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j\" (UID: \"08e6f78f-275f-4778-b3b4-775fa9f0ac84\") " pod="llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j" Nov 04 01:47:19.532250 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:19.532120 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmpkn\" (UniqueName: \"kubernetes.io/projected/08e6f78f-275f-4778-b3b4-775fa9f0ac84-kube-api-access-gmpkn\") pod \"e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j\" (UID: \"08e6f78f-275f-4778-b3b4-775fa9f0ac84\") " pod="llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j" Nov 04 01:47:19.532250 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:19.532151 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08e6f78f-275f-4778-b3b4-775fa9f0ac84-model-cache\") pod \"e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j\" (UID: \"08e6f78f-275f-4778-b3b4-775fa9f0ac84\") " pod="llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j" Nov 04 01:47:19.532340 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:19.532271 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08e6f78f-275f-4778-b3b4-775fa9f0ac84-home\") pod \"e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j\" (UID: \"08e6f78f-275f-4778-b3b4-775fa9f0ac84\") " pod="llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j" Nov 04 01:47:19.532379 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:19.532335 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08e6f78f-275f-4778-b3b4-775fa9f0ac84-tmp-dir\") pod \"e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j\" (UID: \"08e6f78f-275f-4778-b3b4-775fa9f0ac84\") " pod="llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j" Nov 04 01:47:19.532442 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:19.532421 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08e6f78f-275f-4778-b3b4-775fa9f0ac84-dshm\") pod \"e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j\" (UID: \"08e6f78f-275f-4778-b3b4-775fa9f0ac84\") " pod="llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j" Nov 04 01:47:19.633701 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:19.633668 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08e6f78f-275f-4778-b3b4-775fa9f0ac84-tls-certs\") pod \"e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j\" (UID: \"08e6f78f-275f-4778-b3b4-775fa9f0ac84\") " pod="llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j" Nov 04 01:47:19.633883 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:19.633709 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmpkn\" (UniqueName: \"kubernetes.io/projected/08e6f78f-275f-4778-b3b4-775fa9f0ac84-kube-api-access-gmpkn\") pod \"e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j\" (UID: \"08e6f78f-275f-4778-b3b4-775fa9f0ac84\") " pod="llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j" Nov 04 01:47:19.633883 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:19.633740 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08e6f78f-275f-4778-b3b4-775fa9f0ac84-model-cache\") pod \"e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j\" (UID: \"08e6f78f-275f-4778-b3b4-775fa9f0ac84\") " pod="llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j" Nov 04 01:47:19.633883 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:19.633778 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08e6f78f-275f-4778-b3b4-775fa9f0ac84-home\") pod \"e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j\" (UID: \"08e6f78f-275f-4778-b3b4-775fa9f0ac84\") " pod="llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j" Nov 04 01:47:19.633883 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:19.633827 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08e6f78f-275f-4778-b3b4-775fa9f0ac84-tmp-dir\") pod \"e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j\" (UID: \"08e6f78f-275f-4778-b3b4-775fa9f0ac84\") " pod="llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j" Nov 04 01:47:19.634175 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:19.633890 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08e6f78f-275f-4778-b3b4-775fa9f0ac84-dshm\") pod \"e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j\" (UID: \"08e6f78f-275f-4778-b3b4-775fa9f0ac84\") " pod="llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j" Nov 04 01:47:19.634239 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:19.634163 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08e6f78f-275f-4778-b3b4-775fa9f0ac84-model-cache\") pod \"e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j\" (UID: \"08e6f78f-275f-4778-b3b4-775fa9f0ac84\") " pod="llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j" Nov 04 01:47:19.634239 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:19.634182 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08e6f78f-275f-4778-b3b4-775fa9f0ac84-home\") pod \"e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j\" (UID: \"08e6f78f-275f-4778-b3b4-775fa9f0ac84\") " pod="llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j" Nov 04 01:47:19.634361 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:19.634287 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08e6f78f-275f-4778-b3b4-775fa9f0ac84-tmp-dir\") pod \"e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j\" (UID: \"08e6f78f-275f-4778-b3b4-775fa9f0ac84\") " pod="llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j" Nov 04 01:47:19.636320 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:19.636294 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08e6f78f-275f-4778-b3b4-775fa9f0ac84-dshm\") pod \"e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j\" (UID: \"08e6f78f-275f-4778-b3b4-775fa9f0ac84\") " pod="llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j" Nov 04 01:47:19.636432 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:19.636417 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08e6f78f-275f-4778-b3b4-775fa9f0ac84-tls-certs\") pod \"e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j\" (UID: \"08e6f78f-275f-4778-b3b4-775fa9f0ac84\") " pod="llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j" Nov 04 01:47:19.645695 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:19.645673 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmpkn\" (UniqueName: \"kubernetes.io/projected/08e6f78f-275f-4778-b3b4-775fa9f0ac84-kube-api-access-gmpkn\") pod \"e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j\" (UID: \"08e6f78f-275f-4778-b3b4-775fa9f0ac84\") " pod="llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j" Nov 04 01:47:19.738468 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:19.738442 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j" Nov 04 01:47:20.014729 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:20.014703 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j"] Nov 04 01:47:20.016793 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:47:20.016754 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08e6f78f_275f_4778_b3b4_775fa9f0ac84.slice/crio-c83edfc9750a0dcf8459ae75044f849be1c6900416f2d482b3a3fbfef5ca46a4 WatchSource:0}: Error finding container c83edfc9750a0dcf8459ae75044f849be1c6900416f2d482b3a3fbfef5ca46a4: Status 404 returned error can't find the container with id c83edfc9750a0dcf8459ae75044f849be1c6900416f2d482b3a3fbfef5ca46a4 Nov 04 01:47:20.203977 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:20.203940 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j" event={"ID":"08e6f78f-275f-4778-b3b4-775fa9f0ac84","Type":"ContainerStarted","Data":"52e51b33dcbe001ae735248c46db2258f0ee22e1b01429df04b9284545a6ffc4"} Nov 04 01:47:20.203977 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:20.203982 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j" event={"ID":"08e6f78f-275f-4778-b3b4-775fa9f0ac84","Type":"ContainerStarted","Data":"c83edfc9750a0dcf8459ae75044f849be1c6900416f2d482b3a3fbfef5ca46a4"} Nov 04 01:47:20.221810 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:20.221760 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j" podStartSLOduration=1.22174782 podStartE2EDuration="1.22174782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:47:20.220235827 +0000 UTC m=+2925.162084730" watchObservedRunningTime="2026-11-04 01:47:20.22174782 +0000 UTC m=+2925.163596722" Nov 04 01:47:20.739203 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:20.739175 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j" Nov 04 01:47:29.738641 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:29.738601 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j" Nov 04 01:47:29.751237 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:29.751217 7059 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j" Nov 04 01:47:30.279154 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:30.279123 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j" Nov 04 01:47:36.480660 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:47:36.480618 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:47:36.698039 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:36.698004 7059 scope.go:122] "RemoveContainer" containerID="f468a813ddb6f363f9e51fe9f05759ee78f491f0e470bbfe5dac7676ff00a6a5" Nov 04 01:47:50.713787 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:50.713751 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z8s8k"] Nov 04 01:47:50.719832 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:50.719810 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z8s8k" Nov 04 01:47:50.724532 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:50.724506 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:47:50.724675 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:50.724615 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:47:50.726698 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:50.726550 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z8s8k"] Nov 04 01:47:50.818557 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:50.818531 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d93f92c-5a11-4a6e-9424-c5f01e53d280-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-z8s8k\" (UID: \"8d93f92c-5a11-4a6e-9424-c5f01e53d280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z8s8k" Nov 04 01:47:50.818705 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:50.818579 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d93f92c-5a11-4a6e-9424-c5f01e53d280-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-z8s8k\" (UID: \"8d93f92c-5a11-4a6e-9424-c5f01e53d280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z8s8k" Nov 04 01:47:50.818754 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:50.818722 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bwwt\" (UniqueName: \"kubernetes.io/projected/8d93f92c-5a11-4a6e-9424-c5f01e53d280-kube-api-access-2bwwt\") pod \"maas-api-revoke-sub-452b6d7b-z8s8k\" (UID: \"8d93f92c-5a11-4a6e-9424-c5f01e53d280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z8s8k" Nov 04 01:47:50.920169 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:50.920140 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d93f92c-5a11-4a6e-9424-c5f01e53d280-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-z8s8k\" (UID: \"8d93f92c-5a11-4a6e-9424-c5f01e53d280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z8s8k" Nov 04 01:47:50.920334 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:50.920241 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bwwt\" (UniqueName: \"kubernetes.io/projected/8d93f92c-5a11-4a6e-9424-c5f01e53d280-kube-api-access-2bwwt\") pod \"maas-api-revoke-sub-452b6d7b-z8s8k\" (UID: \"8d93f92c-5a11-4a6e-9424-c5f01e53d280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z8s8k" Nov 04 01:47:50.920427 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:50.920352 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d93f92c-5a11-4a6e-9424-c5f01e53d280-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-z8s8k\" (UID: \"8d93f92c-5a11-4a6e-9424-c5f01e53d280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z8s8k" Nov 04 01:47:50.920882 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:50.920855 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d93f92c-5a11-4a6e-9424-c5f01e53d280-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-z8s8k\" (UID: \"8d93f92c-5a11-4a6e-9424-c5f01e53d280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z8s8k" Nov 04 01:47:50.922780 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:50.922758 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d93f92c-5a11-4a6e-9424-c5f01e53d280-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-z8s8k\" (UID: \"8d93f92c-5a11-4a6e-9424-c5f01e53d280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z8s8k" Nov 04 01:47:50.928299 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:50.928278 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bwwt\" (UniqueName: \"kubernetes.io/projected/8d93f92c-5a11-4a6e-9424-c5f01e53d280-kube-api-access-2bwwt\") pod \"maas-api-revoke-sub-452b6d7b-z8s8k\" (UID: \"8d93f92c-5a11-4a6e-9424-c5f01e53d280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z8s8k" Nov 04 01:47:51.035455 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:51.035396 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z8s8k" Nov 04 01:47:51.309449 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:51.309425 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z8s8k"] Nov 04 01:47:51.311134 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:47:51.311108 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d93f92c_5a11_4a6e_9424_c5f01e53d280.slice/crio-63144e308b790e354ae720c28e3572d0f4b28ff2b3d91227c31ec45efe68962a WatchSource:0}: Error finding container 63144e308b790e354ae720c28e3572d0f4b28ff2b3d91227c31ec45efe68962a: Status 404 returned error can't find the container with id 63144e308b790e354ae720c28e3572d0f4b28ff2b3d91227c31ec45efe68962a Nov 04 01:47:51.394435 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:51.394406 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z8s8k" event={"ID":"8d93f92c-5a11-4a6e-9424-c5f01e53d280","Type":"ContainerStarted","Data":"63144e308b790e354ae720c28e3572d0f4b28ff2b3d91227c31ec45efe68962a"} Nov 04 01:47:52.403306 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:52.403265 7059 generic.go:356] "Generic (PLEG): container finished" podID="8d93f92c-5a11-4a6e-9424-c5f01e53d280" containerID="6c2465118474c2babd7ba2797c66beaebdc1be0ba942fc6c20ebf8f07486d0db" exitCode=0 Nov 04 01:47:52.403791 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:52.403339 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z8s8k" event={"ID":"8d93f92c-5a11-4a6e-9424-c5f01e53d280","Type":"ContainerDied","Data":"6c2465118474c2babd7ba2797c66beaebdc1be0ba942fc6c20ebf8f07486d0db"} Nov 04 01:47:53.823351 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:53.823268 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z8s8k" Nov 04 01:47:53.912950 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:53.902062 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d93f92c-5a11-4a6e-9424-c5f01e53d280-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d93f92c-5a11-4a6e-9424-c5f01e53d280-maas-api-xks-ca\") pod \"8d93f92c-5a11-4a6e-9424-c5f01e53d280\" (UID: \"8d93f92c-5a11-4a6e-9424-c5f01e53d280\") " Nov 04 01:47:53.915680 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:53.915653 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d93f92c-5a11-4a6e-9424-c5f01e53d280-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d93f92c-5a11-4a6e-9424-c5f01e53d280-maas-api-openshift-ca\") pod \"8d93f92c-5a11-4a6e-9424-c5f01e53d280\" (UID: \"8d93f92c-5a11-4a6e-9424-c5f01e53d280\") " Nov 04 01:47:53.924161 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:53.924137 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d93f92c-5a11-4a6e-9424-c5f01e53d280-kube-api-access-2bwwt\" (UniqueName: \"kubernetes.io/projected/8d93f92c-5a11-4a6e-9424-c5f01e53d280-kube-api-access-2bwwt\") pod \"8d93f92c-5a11-4a6e-9424-c5f01e53d280\" (UID: \"8d93f92c-5a11-4a6e-9424-c5f01e53d280\") " Nov 04 01:47:53.925088 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:53.921688 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d93f92c-5a11-4a6e-9424-c5f01e53d280-maas-api-openshift-ca" pod "8d93f92c-5a11-4a6e-9424-c5f01e53d280" (UID: "8d93f92c-5a11-4a6e-9424-c5f01e53d280"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:47:53.925235 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:53.924021 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d93f92c-5a11-4a6e-9424-c5f01e53d280-maas-api-xks-ca" pod "8d93f92c-5a11-4a6e-9424-c5f01e53d280" (UID: "8d93f92c-5a11-4a6e-9424-c5f01e53d280"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:47:53.929737 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:53.929712 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d93f92c-5a11-4a6e-9424-c5f01e53d280-kube-api-access-2bwwt" pod "8d93f92c-5a11-4a6e-9424-c5f01e53d280" (UID: "8d93f92c-5a11-4a6e-9424-c5f01e53d280"). InnerVolumeSpecName "kube-api-access-2bwwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:47:54.026380 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:54.026299 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d93f92c-5a11-4a6e-9424-c5f01e53d280-maas-api-xks-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:54.026380 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:54.026333 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d93f92c-5a11-4a6e-9424-c5f01e53d280-maas-api-openshift-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:54.026380 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:54.026343 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bwwt\" (UniqueName: \"kubernetes.io/projected/8d93f92c-5a11-4a6e-9424-c5f01e53d280-kube-api-access-2bwwt\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:54.421568 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:54.421546 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z8s8k" Nov 04 01:47:54.421719 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:54.421568 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z8s8k" event={"ID":"8d93f92c-5a11-4a6e-9424-c5f01e53d280","Type":"ContainerDied","Data":"63144e308b790e354ae720c28e3572d0f4b28ff2b3d91227c31ec45efe68962a"} Nov 04 01:47:54.421719 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:47:54.421613 7059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63144e308b790e354ae720c28e3572d0f4b28ff2b3d91227c31ec45efe68962a" Nov 04 01:48:01.574645 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:01.574603 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j"] Nov 04 01:48:04.067307 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:04.067263 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fp6dv"] Nov 04 01:48:04.068061 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:04.068037 7059 cpu_manager.go:414] "containerMap: removing container" podUID="8d93f92c-5a11-4a6e-9424-c5f01e53d280" containerName="revoke-keys" Nov 04 01:48:04.068199 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:04.068063 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d93f92c-5a11-4a6e-9424-c5f01e53d280" containerName="revoke-keys" Nov 04 01:48:04.068266 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:04.068198 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d93f92c-5a11-4a6e-9424-c5f01e53d280" containerName="revoke-keys" Nov 04 01:48:04.071887 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:04.071864 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fp6dv" Nov 04 01:48:04.082240 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:04.082215 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:48:04.082373 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:04.082262 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:48:04.091909 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:04.091883 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fp6dv"] Nov 04 01:48:04.224053 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:04.224005 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/574bfa6e-f2d8-4283-bde1-56857a57c3f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-fp6dv\" (UID: \"574bfa6e-f2d8-4283-bde1-56857a57c3f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fp6dv" Nov 04 01:48:04.224228 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:04.224087 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdrlt\" (UniqueName: \"kubernetes.io/projected/574bfa6e-f2d8-4283-bde1-56857a57c3f0-kube-api-access-kdrlt\") pod \"maas-api-revoke-sub-3e31a506-fp6dv\" (UID: \"574bfa6e-f2d8-4283-bde1-56857a57c3f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fp6dv" Nov 04 01:48:04.224348 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:04.224249 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/574bfa6e-f2d8-4283-bde1-56857a57c3f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-fp6dv\" (UID: \"574bfa6e-f2d8-4283-bde1-56857a57c3f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fp6dv" Nov 04 01:48:04.325691 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:04.325616 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/574bfa6e-f2d8-4283-bde1-56857a57c3f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-fp6dv\" (UID: \"574bfa6e-f2d8-4283-bde1-56857a57c3f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fp6dv" Nov 04 01:48:04.325847 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:04.325700 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/574bfa6e-f2d8-4283-bde1-56857a57c3f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-fp6dv\" (UID: \"574bfa6e-f2d8-4283-bde1-56857a57c3f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fp6dv" Nov 04 01:48:04.325847 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:04.325778 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdrlt\" (UniqueName: \"kubernetes.io/projected/574bfa6e-f2d8-4283-bde1-56857a57c3f0-kube-api-access-kdrlt\") pod \"maas-api-revoke-sub-3e31a506-fp6dv\" (UID: \"574bfa6e-f2d8-4283-bde1-56857a57c3f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fp6dv" Nov 04 01:48:04.326575 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:04.326486 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/574bfa6e-f2d8-4283-bde1-56857a57c3f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-fp6dv\" (UID: \"574bfa6e-f2d8-4283-bde1-56857a57c3f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fp6dv" Nov 04 01:48:04.328403 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:04.328370 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/574bfa6e-f2d8-4283-bde1-56857a57c3f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-fp6dv\" (UID: \"574bfa6e-f2d8-4283-bde1-56857a57c3f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fp6dv" Nov 04 01:48:04.336075 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:04.336049 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdrlt\" (UniqueName: \"kubernetes.io/projected/574bfa6e-f2d8-4283-bde1-56857a57c3f0-kube-api-access-kdrlt\") pod \"maas-api-revoke-sub-3e31a506-fp6dv\" (UID: \"574bfa6e-f2d8-4283-bde1-56857a57c3f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fp6dv" Nov 04 01:48:04.390795 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:04.390772 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fp6dv" Nov 04 01:48:04.713954 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:04.711151 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fp6dv"] Nov 04 01:48:05.512068 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:05.511047 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fp6dv" event={"ID":"574bfa6e-f2d8-4283-bde1-56857a57c3f0","Type":"ContainerStarted","Data":"1dba213e3fecefc9ed6869dac8804581ab81cc156102dc73efcb8624957e0374"} Nov 04 01:48:05.512068 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:05.511113 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fp6dv" event={"ID":"574bfa6e-f2d8-4283-bde1-56857a57c3f0","Type":"ContainerStarted","Data":"ac4171b7fd738a8a0c83c790027462643d47ace9371a0572e273affc06c04060"} Nov 04 01:48:05.536241 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:05.536191 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fp6dv" podStartSLOduration=1.536176592 podStartE2EDuration="1.536176592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:48:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:48:05.533023155 +0000 UTC m=+2970.474872058" watchObservedRunningTime="2026-11-04 01:48:05.536176592 +0000 UTC m=+2970.478025495" Nov 04 01:48:06.523411 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:06.523310 7059 generic.go:356] "Generic (PLEG): container finished" podID="574bfa6e-f2d8-4283-bde1-56857a57c3f0" containerID="1dba213e3fecefc9ed6869dac8804581ab81cc156102dc73efcb8624957e0374" exitCode=0 Nov 04 01:48:06.523808 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:06.523400 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fp6dv" event={"ID":"574bfa6e-f2d8-4283-bde1-56857a57c3f0","Type":"ContainerDied","Data":"1dba213e3fecefc9ed6869dac8804581ab81cc156102dc73efcb8624957e0374"} Nov 04 01:48:06.532839 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:48:06.532808 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:48:07.703101 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:07.703067 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fp6dv" Nov 04 01:48:07.826282 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:07.826178 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/574bfa6e-f2d8-4283-bde1-56857a57c3f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/574bfa6e-f2d8-4283-bde1-56857a57c3f0-maas-api-openshift-ca\") pod \"574bfa6e-f2d8-4283-bde1-56857a57c3f0\" (UID: \"574bfa6e-f2d8-4283-bde1-56857a57c3f0\") " Nov 04 01:48:07.826282 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:07.826260 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/574bfa6e-f2d8-4283-bde1-56857a57c3f0-kube-api-access-kdrlt\" (UniqueName: \"kubernetes.io/projected/574bfa6e-f2d8-4283-bde1-56857a57c3f0-kube-api-access-kdrlt\") pod \"574bfa6e-f2d8-4283-bde1-56857a57c3f0\" (UID: \"574bfa6e-f2d8-4283-bde1-56857a57c3f0\") " Nov 04 01:48:07.826519 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:07.826364 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/574bfa6e-f2d8-4283-bde1-56857a57c3f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/574bfa6e-f2d8-4283-bde1-56857a57c3f0-maas-api-xks-ca\") pod \"574bfa6e-f2d8-4283-bde1-56857a57c3f0\" (UID: \"574bfa6e-f2d8-4283-bde1-56857a57c3f0\") " Nov 04 01:48:07.826693 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:07.826666 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/574bfa6e-f2d8-4283-bde1-56857a57c3f0-maas-api-openshift-ca" pod "574bfa6e-f2d8-4283-bde1-56857a57c3f0" (UID: "574bfa6e-f2d8-4283-bde1-56857a57c3f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:48:07.826791 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:07.826771 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/574bfa6e-f2d8-4283-bde1-56857a57c3f0-maas-api-openshift-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:07.829178 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:07.829139 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/574bfa6e-f2d8-4283-bde1-56857a57c3f0-kube-api-access-kdrlt" pod "574bfa6e-f2d8-4283-bde1-56857a57c3f0" (UID: "574bfa6e-f2d8-4283-bde1-56857a57c3f0"). InnerVolumeSpecName "kube-api-access-kdrlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:48:07.830069 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:07.830023 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/574bfa6e-f2d8-4283-bde1-56857a57c3f0-maas-api-xks-ca" pod "574bfa6e-f2d8-4283-bde1-56857a57c3f0" (UID: "574bfa6e-f2d8-4283-bde1-56857a57c3f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:48:07.928098 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:07.928060 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdrlt\" (UniqueName: \"kubernetes.io/projected/574bfa6e-f2d8-4283-bde1-56857a57c3f0-kube-api-access-kdrlt\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:07.928098 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:07.928098 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/574bfa6e-f2d8-4283-bde1-56857a57c3f0-maas-api-xks-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:08.025212 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:08.025179 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzrrd"] Nov 04 01:48:08.028960 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:08.028911 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qzrrd"] Nov 04 01:48:08.537405 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:08.537366 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fp6dv" event={"ID":"574bfa6e-f2d8-4283-bde1-56857a57c3f0","Type":"ContainerDied","Data":"ac4171b7fd738a8a0c83c790027462643d47ace9371a0572e273affc06c04060"} Nov 04 01:48:08.537405 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:08.537398 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fp6dv" Nov 04 01:48:08.537671 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:08.537416 7059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac4171b7fd738a8a0c83c790027462643d47ace9371a0572e273affc06c04060" Nov 04 01:48:09.204986 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:09.197038 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05f0273d-e8a3-49c9-91b7-0f4fcf2a564f" path="/var/lib/kubelet/pods/05f0273d-e8a3-49c9-91b7-0f4fcf2a564f/volumes" Nov 04 01:48:16.601455 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:16.601251 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j" podUID="08e6f78f-275f-4778-b3b4-775fa9f0ac84" containerName="main" containerID="cri-o://52e51b33dcbe001ae735248c46db2258f0ee22e1b01429df04b9284545a6ffc4" gracePeriod=45 Nov 04 01:48:17.088948 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:17.051053 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j" Nov 04 01:48:17.124970 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:17.121547 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08e6f78f-275f-4778-b3b4-775fa9f0ac84-kube-api-access-gmpkn\" (UniqueName: \"kubernetes.io/projected/08e6f78f-275f-4778-b3b4-775fa9f0ac84-kube-api-access-gmpkn\") pod \"08e6f78f-275f-4778-b3b4-775fa9f0ac84\" (UID: \"08e6f78f-275f-4778-b3b4-775fa9f0ac84\") " Nov 04 01:48:17.124970 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:17.121601 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08e6f78f-275f-4778-b3b4-775fa9f0ac84-tls-certs\" (UniqueName: \"kubernetes.io/secret/08e6f78f-275f-4778-b3b4-775fa9f0ac84-tls-certs\") pod \"08e6f78f-275f-4778-b3b4-775fa9f0ac84\" (UID: \"08e6f78f-275f-4778-b3b4-775fa9f0ac84\") " Nov 04 01:48:17.124970 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:17.121645 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08e6f78f-275f-4778-b3b4-775fa9f0ac84-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08e6f78f-275f-4778-b3b4-775fa9f0ac84-tmp-dir\") pod \"08e6f78f-275f-4778-b3b4-775fa9f0ac84\" (UID: \"08e6f78f-275f-4778-b3b4-775fa9f0ac84\") " Nov 04 01:48:17.124970 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:17.121696 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08e6f78f-275f-4778-b3b4-775fa9f0ac84-home\" (UniqueName: \"kubernetes.io/empty-dir/08e6f78f-275f-4778-b3b4-775fa9f0ac84-home\") pod \"08e6f78f-275f-4778-b3b4-775fa9f0ac84\" (UID: \"08e6f78f-275f-4778-b3b4-775fa9f0ac84\") " Nov 04 01:48:17.124970 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:17.121790 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08e6f78f-275f-4778-b3b4-775fa9f0ac84-dshm\" (UniqueName: \"kubernetes.io/empty-dir/08e6f78f-275f-4778-b3b4-775fa9f0ac84-dshm\") pod \"08e6f78f-275f-4778-b3b4-775fa9f0ac84\" (UID: \"08e6f78f-275f-4778-b3b4-775fa9f0ac84\") " Nov 04 01:48:17.124970 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:17.121841 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08e6f78f-275f-4778-b3b4-775fa9f0ac84-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08e6f78f-275f-4778-b3b4-775fa9f0ac84-model-cache\") pod \"08e6f78f-275f-4778-b3b4-775fa9f0ac84\" (UID: \"08e6f78f-275f-4778-b3b4-775fa9f0ac84\") " Nov 04 01:48:17.124970 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:17.122581 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08e6f78f-275f-4778-b3b4-775fa9f0ac84-model-cache" pod "08e6f78f-275f-4778-b3b4-775fa9f0ac84" (UID: "08e6f78f-275f-4778-b3b4-775fa9f0ac84"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:48:17.124970 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:17.123151 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08e6f78f-275f-4778-b3b4-775fa9f0ac84-tmp-dir" pod "08e6f78f-275f-4778-b3b4-775fa9f0ac84" (UID: "08e6f78f-275f-4778-b3b4-775fa9f0ac84"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:48:17.124970 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:17.123637 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08e6f78f-275f-4778-b3b4-775fa9f0ac84-home" pod "08e6f78f-275f-4778-b3b4-775fa9f0ac84" (UID: "08e6f78f-275f-4778-b3b4-775fa9f0ac84"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:48:17.126169 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:17.126136 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08e6f78f-275f-4778-b3b4-775fa9f0ac84-kube-api-access-gmpkn" pod "08e6f78f-275f-4778-b3b4-775fa9f0ac84" (UID: "08e6f78f-275f-4778-b3b4-775fa9f0ac84"). InnerVolumeSpecName "kube-api-access-gmpkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:48:17.126303 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:17.126274 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08e6f78f-275f-4778-b3b4-775fa9f0ac84-dshm" pod "08e6f78f-275f-4778-b3b4-775fa9f0ac84" (UID: "08e6f78f-275f-4778-b3b4-775fa9f0ac84"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:48:17.127090 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:17.127066 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08e6f78f-275f-4778-b3b4-775fa9f0ac84-tls-certs" pod "08e6f78f-275f-4778-b3b4-775fa9f0ac84" (UID: "08e6f78f-275f-4778-b3b4-775fa9f0ac84"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:48:17.223366 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:17.223320 7059 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08e6f78f-275f-4778-b3b4-775fa9f0ac84-tmp-dir\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:17.223366 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:17.223355 7059 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08e6f78f-275f-4778-b3b4-775fa9f0ac84-home\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:17.223366 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:17.223371 7059 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08e6f78f-275f-4778-b3b4-775fa9f0ac84-dshm\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:17.223702 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:17.223384 7059 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08e6f78f-275f-4778-b3b4-775fa9f0ac84-model-cache\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:17.223702 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:17.223402 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmpkn\" (UniqueName: \"kubernetes.io/projected/08e6f78f-275f-4778-b3b4-775fa9f0ac84-kube-api-access-gmpkn\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:17.223702 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:17.223418 7059 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08e6f78f-275f-4778-b3b4-775fa9f0ac84-tls-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:17.624221 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:17.624188 7059 generic.go:356] "Generic (PLEG): container finished" podID="08e6f78f-275f-4778-b3b4-775fa9f0ac84" containerID="52e51b33dcbe001ae735248c46db2258f0ee22e1b01429df04b9284545a6ffc4" exitCode=0 Nov 04 01:48:17.624720 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:17.624242 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j" Nov 04 01:48:17.624720 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:17.624265 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j" event={"ID":"08e6f78f-275f-4778-b3b4-775fa9f0ac84","Type":"ContainerDied","Data":"52e51b33dcbe001ae735248c46db2258f0ee22e1b01429df04b9284545a6ffc4"} Nov 04 01:48:17.624720 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:17.624306 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j" event={"ID":"08e6f78f-275f-4778-b3b4-775fa9f0ac84","Type":"ContainerDied","Data":"c83edfc9750a0dcf8459ae75044f849be1c6900416f2d482b3a3fbfef5ca46a4"} Nov 04 01:48:17.624720 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:17.624322 7059 scope.go:122] "RemoveContainer" containerID="52e51b33dcbe001ae735248c46db2258f0ee22e1b01429df04b9284545a6ffc4" Nov 04 01:48:17.637531 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:17.637514 7059 scope.go:122] "RemoveContainer" containerID="52e51b33dcbe001ae735248c46db2258f0ee22e1b01429df04b9284545a6ffc4" Nov 04 01:48:17.637819 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:48:17.637802 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52e51b33dcbe001ae735248c46db2258f0ee22e1b01429df04b9284545a6ffc4\": container with ID starting with 52e51b33dcbe001ae735248c46db2258f0ee22e1b01429df04b9284545a6ffc4 not found: ID does not exist" containerID="52e51b33dcbe001ae735248c46db2258f0ee22e1b01429df04b9284545a6ffc4" Nov 04 01:48:17.637869 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:17.637828 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52e51b33dcbe001ae735248c46db2258f0ee22e1b01429df04b9284545a6ffc4"} err="failed to get container status \"52e51b33dcbe001ae735248c46db2258f0ee22e1b01429df04b9284545a6ffc4\": rpc error: code = NotFound desc = could not find container \"52e51b33dcbe001ae735248c46db2258f0ee22e1b01429df04b9284545a6ffc4\": container with ID starting with 52e51b33dcbe001ae735248c46db2258f0ee22e1b01429df04b9284545a6ffc4 not found: ID does not exist" Nov 04 01:48:17.647208 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:17.647186 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j"] Nov 04 01:48:17.650456 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:17.650436 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-eab16c40-kserve-86f8c4477-2mt5j"] Nov 04 01:48:19.194390 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:19.194352 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08e6f78f-275f-4778-b3b4-775fa9f0ac84" path="/var/lib/kubelet/pods/08e6f78f-275f-4778-b3b4-775fa9f0ac84/volumes" Nov 04 01:48:22.432838 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:22.432805 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ced87dd-45r56"] Nov 04 01:48:22.433350 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:22.433335 7059 cpu_manager.go:414] "containerMap: removing container" podUID="08e6f78f-275f-4778-b3b4-775fa9f0ac84" containerName="main" Nov 04 01:48:22.433402 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:22.433352 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08e6f78f-275f-4778-b3b4-775fa9f0ac84" containerName="main" Nov 04 01:48:22.433402 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:22.433371 7059 cpu_manager.go:414] "containerMap: removing container" podUID="574bfa6e-f2d8-4283-bde1-56857a57c3f0" containerName="revoke-keys" Nov 04 01:48:22.433402 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:22.433377 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="574bfa6e-f2d8-4283-bde1-56857a57c3f0" containerName="revoke-keys" Nov 04 01:48:22.433516 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:22.433448 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="08e6f78f-275f-4778-b3b4-775fa9f0ac84" containerName="main" Nov 04 01:48:22.433516 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:22.433456 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="574bfa6e-f2d8-4283-bde1-56857a57c3f0" containerName="revoke-keys" Nov 04 01:48:22.438030 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:22.438013 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ced87dd-45r56" Nov 04 01:48:22.446977 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:22.446950 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:48:22.447140 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:22.447078 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:48:22.449567 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:22.449534 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ced87dd-45r56"] Nov 04 01:48:22.517499 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:22.517471 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqzzx\" (UniqueName: \"kubernetes.io/projected/19c1d045-b17f-4a30-abbf-473456346eb7-kube-api-access-vqzzx\") pod \"maas-api-revoke-sub-2ced87dd-45r56\" (UID: \"19c1d045-b17f-4a30-abbf-473456346eb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ced87dd-45r56" Nov 04 01:48:22.517608 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:22.517589 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19c1d045-b17f-4a30-abbf-473456346eb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ced87dd-45r56\" (UID: \"19c1d045-b17f-4a30-abbf-473456346eb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ced87dd-45r56" Nov 04 01:48:22.517666 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:22.517655 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19c1d045-b17f-4a30-abbf-473456346eb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ced87dd-45r56\" (UID: \"19c1d045-b17f-4a30-abbf-473456346eb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ced87dd-45r56" Nov 04 01:48:22.619049 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:22.619010 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19c1d045-b17f-4a30-abbf-473456346eb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ced87dd-45r56\" (UID: \"19c1d045-b17f-4a30-abbf-473456346eb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ced87dd-45r56" Nov 04 01:48:22.619189 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:22.619079 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19c1d045-b17f-4a30-abbf-473456346eb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ced87dd-45r56\" (UID: \"19c1d045-b17f-4a30-abbf-473456346eb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ced87dd-45r56" Nov 04 01:48:22.619189 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:22.619130 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqzzx\" (UniqueName: \"kubernetes.io/projected/19c1d045-b17f-4a30-abbf-473456346eb7-kube-api-access-vqzzx\") pod \"maas-api-revoke-sub-2ced87dd-45r56\" (UID: \"19c1d045-b17f-4a30-abbf-473456346eb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ced87dd-45r56" Nov 04 01:48:22.619641 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:22.619619 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19c1d045-b17f-4a30-abbf-473456346eb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ced87dd-45r56\" (UID: \"19c1d045-b17f-4a30-abbf-473456346eb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ced87dd-45r56" Nov 04 01:48:22.621517 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:22.621493 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19c1d045-b17f-4a30-abbf-473456346eb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ced87dd-45r56\" (UID: \"19c1d045-b17f-4a30-abbf-473456346eb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ced87dd-45r56" Nov 04 01:48:22.631718 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:22.631695 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqzzx\" (UniqueName: \"kubernetes.io/projected/19c1d045-b17f-4a30-abbf-473456346eb7-kube-api-access-vqzzx\") pod \"maas-api-revoke-sub-2ced87dd-45r56\" (UID: \"19c1d045-b17f-4a30-abbf-473456346eb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ced87dd-45r56" Nov 04 01:48:22.754694 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:22.754641 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ced87dd-45r56" Nov 04 01:48:23.061090 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:23.061061 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ced87dd-45r56"] Nov 04 01:48:23.061497 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:48:23.061478 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c1d045_b17f_4a30_abbf_473456346eb7.slice/crio-4c88e0614bbb8a73ef53ad49f751203fd469b176fb5aa33590b2719a224ff0c9 WatchSource:0}: Error finding container 4c88e0614bbb8a73ef53ad49f751203fd469b176fb5aa33590b2719a224ff0c9: Status 404 returned error can't find the container with id 4c88e0614bbb8a73ef53ad49f751203fd469b176fb5aa33590b2719a224ff0c9 Nov 04 01:48:23.685255 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:23.685212 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ced87dd-45r56" event={"ID":"19c1d045-b17f-4a30-abbf-473456346eb7","Type":"ContainerStarted","Data":"daae62ccdd6688ab29fbfd45578c35d8a9562c726bcb9302be6087391657c1fd"} Nov 04 01:48:23.685643 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:23.685277 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ced87dd-45r56" event={"ID":"19c1d045-b17f-4a30-abbf-473456346eb7","Type":"ContainerStarted","Data":"4c88e0614bbb8a73ef53ad49f751203fd469b176fb5aa33590b2719a224ff0c9"} Nov 04 01:48:23.714167 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:23.714126 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ced87dd-45r56" podStartSLOduration=1.71411308 podStartE2EDuration="1.71411308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:48:23.712792405 +0000 UTC m=+2988.654641287" watchObservedRunningTime="2026-11-04 01:48:23.71411308 +0000 UTC m=+2988.655961983" Nov 04 01:48:24.700059 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:24.700020 7059 generic.go:356] "Generic (PLEG): container finished" podID="19c1d045-b17f-4a30-abbf-473456346eb7" containerID="daae62ccdd6688ab29fbfd45578c35d8a9562c726bcb9302be6087391657c1fd" exitCode=0 Nov 04 01:48:24.700430 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:24.700106 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ced87dd-45r56" event={"ID":"19c1d045-b17f-4a30-abbf-473456346eb7","Type":"ContainerDied","Data":"daae62ccdd6688ab29fbfd45578c35d8a9562c726bcb9302be6087391657c1fd"} Nov 04 01:48:25.867940 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:25.867895 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ced87dd-45r56" Nov 04 01:48:25.952218 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:25.952190 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19c1d045-b17f-4a30-abbf-473456346eb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19c1d045-b17f-4a30-abbf-473456346eb7-maas-api-xks-ca\") pod \"19c1d045-b17f-4a30-abbf-473456346eb7\" (UID: \"19c1d045-b17f-4a30-abbf-473456346eb7\") " Nov 04 01:48:25.952383 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:25.952230 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19c1d045-b17f-4a30-abbf-473456346eb7-kube-api-access-vqzzx\" (UniqueName: \"kubernetes.io/projected/19c1d045-b17f-4a30-abbf-473456346eb7-kube-api-access-vqzzx\") pod \"19c1d045-b17f-4a30-abbf-473456346eb7\" (UID: \"19c1d045-b17f-4a30-abbf-473456346eb7\") " Nov 04 01:48:25.952449 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:25.952371 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19c1d045-b17f-4a30-abbf-473456346eb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19c1d045-b17f-4a30-abbf-473456346eb7-maas-api-openshift-ca\") pod \"19c1d045-b17f-4a30-abbf-473456346eb7\" (UID: \"19c1d045-b17f-4a30-abbf-473456346eb7\") " Nov 04 01:48:25.952695 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:25.952671 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19c1d045-b17f-4a30-abbf-473456346eb7-maas-api-openshift-ca" pod "19c1d045-b17f-4a30-abbf-473456346eb7" (UID: "19c1d045-b17f-4a30-abbf-473456346eb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:48:25.952815 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:25.952796 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19c1d045-b17f-4a30-abbf-473456346eb7-maas-api-openshift-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:25.954504 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:25.954483 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19c1d045-b17f-4a30-abbf-473456346eb7-maas-api-xks-ca" pod "19c1d045-b17f-4a30-abbf-473456346eb7" (UID: "19c1d045-b17f-4a30-abbf-473456346eb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:48:25.954593 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:25.954570 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c1d045-b17f-4a30-abbf-473456346eb7-kube-api-access-vqzzx" pod "19c1d045-b17f-4a30-abbf-473456346eb7" (UID: "19c1d045-b17f-4a30-abbf-473456346eb7"). InnerVolumeSpecName "kube-api-access-vqzzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:48:26.053980 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:26.053954 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19c1d045-b17f-4a30-abbf-473456346eb7-maas-api-xks-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:26.053980 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:26.053979 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqzzx\" (UniqueName: \"kubernetes.io/projected/19c1d045-b17f-4a30-abbf-473456346eb7-kube-api-access-vqzzx\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:26.713473 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:26.713441 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ced87dd-45r56" event={"ID":"19c1d045-b17f-4a30-abbf-473456346eb7","Type":"ContainerDied","Data":"4c88e0614bbb8a73ef53ad49f751203fd469b176fb5aa33590b2719a224ff0c9"} Nov 04 01:48:26.713666 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:26.713495 7059 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c88e0614bbb8a73ef53ad49f751203fd469b176fb5aa33590b2719a224ff0c9" Nov 04 01:48:26.713666 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:26.713458 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ced87dd-45r56" Nov 04 01:48:34.631498 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.631464 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-847bf86b74-76hwj"] Nov 04 01:48:34.632050 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.632026 7059 cpu_manager.go:414] "containerMap: removing container" podUID="19c1d045-b17f-4a30-abbf-473456346eb7" containerName="revoke-keys" Nov 04 01:48:34.632050 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.632043 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19c1d045-b17f-4a30-abbf-473456346eb7" containerName="revoke-keys" Nov 04 01:48:34.632209 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.632126 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="19c1d045-b17f-4a30-abbf-473456346eb7" containerName="revoke-keys" Nov 04 01:48:34.635404 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.635387 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-847bf86b74-76hwj" Nov 04 01:48:34.644550 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.644527 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 04 01:48:34.644800 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.644744 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nb79d\"" Nov 04 01:48:34.644983 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.644822 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 04 01:48:34.667390 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.667359 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-845d896dc4-gllnl"] Nov 04 01:48:34.671377 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.671355 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-847bf86b74-76hwj"] Nov 04 01:48:34.671495 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.671448 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-845d896dc4-gllnl" Nov 04 01:48:34.689509 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.689488 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-845d896dc4-gllnl"] Nov 04 01:48:34.732482 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.732447 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e503a527-6c89-4264-bfd3-211f6d9d9dfd-openshift-service-ca\") pod \"maas-api-847bf86b74-76hwj\" (UID: \"e503a527-6c89-4264-bfd3-211f6d9d9dfd\") " pod="odh-ai-gateway-infra/maas-api-847bf86b74-76hwj" Nov 04 01:48:34.732643 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.732526 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e503a527-6c89-4264-bfd3-211f6d9d9dfd-metrics-certs\") pod \"maas-api-847bf86b74-76hwj\" (UID: \"e503a527-6c89-4264-bfd3-211f6d9d9dfd\") " pod="odh-ai-gateway-infra/maas-api-847bf86b74-76hwj" Nov 04 01:48:34.732643 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.732558 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e503a527-6c89-4264-bfd3-211f6d9d9dfd-maas-api-tls\") pod \"maas-api-847bf86b74-76hwj\" (UID: \"e503a527-6c89-4264-bfd3-211f6d9d9dfd\") " pod="odh-ai-gateway-infra/maas-api-847bf86b74-76hwj" Nov 04 01:48:34.732643 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.732577 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8kdt\" (UniqueName: \"kubernetes.io/projected/e503a527-6c89-4264-bfd3-211f6d9d9dfd-kube-api-access-v8kdt\") pod \"maas-api-847bf86b74-76hwj\" (UID: \"e503a527-6c89-4264-bfd3-211f6d9d9dfd\") " pod="odh-ai-gateway-infra/maas-api-847bf86b74-76hwj" Nov 04 01:48:34.833316 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.833287 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc928\" (UniqueName: \"kubernetes.io/projected/93d9777a-32e4-4e1d-9231-78f201654998-kube-api-access-hc928\") pod \"maas-api-845d896dc4-gllnl\" (UID: \"93d9777a-32e4-4e1d-9231-78f201654998\") " pod="odh-ai-gateway-infra/maas-api-845d896dc4-gllnl" Nov 04 01:48:34.833491 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.833342 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e503a527-6c89-4264-bfd3-211f6d9d9dfd-metrics-certs\") pod \"maas-api-847bf86b74-76hwj\" (UID: \"e503a527-6c89-4264-bfd3-211f6d9d9dfd\") " pod="odh-ai-gateway-infra/maas-api-847bf86b74-76hwj" Nov 04 01:48:34.833491 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.833421 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e503a527-6c89-4264-bfd3-211f6d9d9dfd-maas-api-tls\") pod \"maas-api-847bf86b74-76hwj\" (UID: \"e503a527-6c89-4264-bfd3-211f6d9d9dfd\") " pod="odh-ai-gateway-infra/maas-api-847bf86b74-76hwj" Nov 04 01:48:34.833491 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.833454 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8kdt\" (UniqueName: \"kubernetes.io/projected/e503a527-6c89-4264-bfd3-211f6d9d9dfd-kube-api-access-v8kdt\") pod \"maas-api-847bf86b74-76hwj\" (UID: \"e503a527-6c89-4264-bfd3-211f6d9d9dfd\") " pod="odh-ai-gateway-infra/maas-api-847bf86b74-76hwj" Nov 04 01:48:34.833611 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.833505 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93d9777a-32e4-4e1d-9231-78f201654998-maas-api-tls\") pod \"maas-api-845d896dc4-gllnl\" (UID: \"93d9777a-32e4-4e1d-9231-78f201654998\") " pod="odh-ai-gateway-infra/maas-api-845d896dc4-gllnl" Nov 04 01:48:34.833611 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.833572 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e503a527-6c89-4264-bfd3-211f6d9d9dfd-openshift-service-ca\") pod \"maas-api-847bf86b74-76hwj\" (UID: \"e503a527-6c89-4264-bfd3-211f6d9d9dfd\") " pod="odh-ai-gateway-infra/maas-api-847bf86b74-76hwj" Nov 04 01:48:34.833695 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.833652 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93d9777a-32e4-4e1d-9231-78f201654998-metrics-certs\") pod \"maas-api-845d896dc4-gllnl\" (UID: \"93d9777a-32e4-4e1d-9231-78f201654998\") " pod="odh-ai-gateway-infra/maas-api-845d896dc4-gllnl" Nov 04 01:48:34.833745 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.833686 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93d9777a-32e4-4e1d-9231-78f201654998-openshift-service-ca\") pod \"maas-api-845d896dc4-gllnl\" (UID: \"93d9777a-32e4-4e1d-9231-78f201654998\") " pod="odh-ai-gateway-infra/maas-api-845d896dc4-gllnl" Nov 04 01:48:34.834294 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.834218 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e503a527-6c89-4264-bfd3-211f6d9d9dfd-openshift-service-ca\") pod \"maas-api-847bf86b74-76hwj\" (UID: \"e503a527-6c89-4264-bfd3-211f6d9d9dfd\") " pod="odh-ai-gateway-infra/maas-api-847bf86b74-76hwj" Nov 04 01:48:34.836086 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.836064 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e503a527-6c89-4264-bfd3-211f6d9d9dfd-metrics-certs\") pod \"maas-api-847bf86b74-76hwj\" (UID: \"e503a527-6c89-4264-bfd3-211f6d9d9dfd\") " pod="odh-ai-gateway-infra/maas-api-847bf86b74-76hwj" Nov 04 01:48:34.836185 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.836114 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e503a527-6c89-4264-bfd3-211f6d9d9dfd-maas-api-tls\") pod \"maas-api-847bf86b74-76hwj\" (UID: \"e503a527-6c89-4264-bfd3-211f6d9d9dfd\") " pod="odh-ai-gateway-infra/maas-api-847bf86b74-76hwj" Nov 04 01:48:34.843017 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.842995 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8kdt\" (UniqueName: \"kubernetes.io/projected/e503a527-6c89-4264-bfd3-211f6d9d9dfd-kube-api-access-v8kdt\") pod \"maas-api-847bf86b74-76hwj\" (UID: \"e503a527-6c89-4264-bfd3-211f6d9d9dfd\") " pod="odh-ai-gateway-infra/maas-api-847bf86b74-76hwj" Nov 04 01:48:34.934989 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.934960 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93d9777a-32e4-4e1d-9231-78f201654998-metrics-certs\") pod \"maas-api-845d896dc4-gllnl\" (UID: \"93d9777a-32e4-4e1d-9231-78f201654998\") " pod="odh-ai-gateway-infra/maas-api-845d896dc4-gllnl" Nov 04 01:48:34.935134 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.934996 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93d9777a-32e4-4e1d-9231-78f201654998-openshift-service-ca\") pod \"maas-api-845d896dc4-gllnl\" (UID: \"93d9777a-32e4-4e1d-9231-78f201654998\") " pod="odh-ai-gateway-infra/maas-api-845d896dc4-gllnl" Nov 04 01:48:34.935134 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.935026 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc928\" (UniqueName: \"kubernetes.io/projected/93d9777a-32e4-4e1d-9231-78f201654998-kube-api-access-hc928\") pod \"maas-api-845d896dc4-gllnl\" (UID: \"93d9777a-32e4-4e1d-9231-78f201654998\") " pod="odh-ai-gateway-infra/maas-api-845d896dc4-gllnl" Nov 04 01:48:34.935134 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.935100 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93d9777a-32e4-4e1d-9231-78f201654998-maas-api-tls\") pod \"maas-api-845d896dc4-gllnl\" (UID: \"93d9777a-32e4-4e1d-9231-78f201654998\") " pod="odh-ai-gateway-infra/maas-api-845d896dc4-gllnl" Nov 04 01:48:34.935581 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.935560 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93d9777a-32e4-4e1d-9231-78f201654998-openshift-service-ca\") pod \"maas-api-845d896dc4-gllnl\" (UID: \"93d9777a-32e4-4e1d-9231-78f201654998\") " pod="odh-ai-gateway-infra/maas-api-845d896dc4-gllnl" Nov 04 01:48:34.937533 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.937511 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93d9777a-32e4-4e1d-9231-78f201654998-maas-api-tls\") pod \"maas-api-845d896dc4-gllnl\" (UID: \"93d9777a-32e4-4e1d-9231-78f201654998\") " pod="odh-ai-gateway-infra/maas-api-845d896dc4-gllnl" Nov 04 01:48:34.937533 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.937528 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93d9777a-32e4-4e1d-9231-78f201654998-metrics-certs\") pod \"maas-api-845d896dc4-gllnl\" (UID: \"93d9777a-32e4-4e1d-9231-78f201654998\") " pod="odh-ai-gateway-infra/maas-api-845d896dc4-gllnl" Nov 04 01:48:34.944150 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.944127 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc928\" (UniqueName: \"kubernetes.io/projected/93d9777a-32e4-4e1d-9231-78f201654998-kube-api-access-hc928\") pod \"maas-api-845d896dc4-gllnl\" (UID: \"93d9777a-32e4-4e1d-9231-78f201654998\") " pod="odh-ai-gateway-infra/maas-api-845d896dc4-gllnl" Nov 04 01:48:34.949995 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.949975 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-847bf86b74-76hwj" Nov 04 01:48:34.991296 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:34.991264 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-845d896dc4-gllnl" Nov 04 01:48:35.242625 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:35.242600 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-847bf86b74-76hwj"] Nov 04 01:48:35.244739 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:48:35.244714 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode503a527_6c89_4264_bfd3_211f6d9d9dfd.slice/crio-edfdd36a6c066a7b35a4a1c7b50587dd610f973fe6fe14bda471a0f5c7f83ebf WatchSource:0}: Error finding container edfdd36a6c066a7b35a4a1c7b50587dd610f973fe6fe14bda471a0f5c7f83ebf: Status 404 returned error can't find the container with id edfdd36a6c066a7b35a4a1c7b50587dd610f973fe6fe14bda471a0f5c7f83ebf Nov 04 01:48:35.313518 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:35.313487 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-845d896dc4-gllnl"] Nov 04 01:48:35.314507 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:48:35.314481 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93d9777a_32e4_4e1d_9231_78f201654998.slice/crio-e9b89ebe86e396d60b86fd3dfd510637ef8faed9d44ce69034313eef1fab46bf WatchSource:0}: Error finding container e9b89ebe86e396d60b86fd3dfd510637ef8faed9d44ce69034313eef1fab46bf: Status 404 returned error can't find the container with id e9b89ebe86e396d60b86fd3dfd510637ef8faed9d44ce69034313eef1fab46bf Nov 04 01:48:35.765835 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:35.765801 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-847bf86b74-76hwj" event={"ID":"e503a527-6c89-4264-bfd3-211f6d9d9dfd","Type":"ContainerStarted","Data":"edfdd36a6c066a7b35a4a1c7b50587dd610f973fe6fe14bda471a0f5c7f83ebf"} Nov 04 01:48:35.767199 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:35.767165 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-845d896dc4-gllnl" event={"ID":"93d9777a-32e4-4e1d-9231-78f201654998","Type":"ContainerStarted","Data":"e9b89ebe86e396d60b86fd3dfd510637ef8faed9d44ce69034313eef1fab46bf"} Nov 04 01:48:36.480660 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:48:36.480617 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:48:36.774661 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:36.774619 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-847bf86b74-76hwj" event={"ID":"e503a527-6c89-4264-bfd3-211f6d9d9dfd","Type":"ContainerStarted","Data":"b320754bba17dc83805a74c37c597cbcf78766a552034cc78154e6921f76c3f8"} Nov 04 01:48:36.775097 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:36.774696 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-847bf86b74-76hwj" Nov 04 01:48:36.776266 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:36.776243 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-845d896dc4-gllnl" event={"ID":"93d9777a-32e4-4e1d-9231-78f201654998","Type":"ContainerStarted","Data":"07b4ef47d8af4b370eb72db5c247b935e95fdcb13ad935d1648245f8e34d02a9"} Nov 04 01:48:36.776354 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:36.776304 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-845d896dc4-gllnl" Nov 04 01:48:36.795549 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:36.795508 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-847bf86b74-76hwj" podStartSLOduration=2.128747758 podStartE2EDuration="2.795495468s" totalImagesPullingTime="666.74771ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:48:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:48:36.793248397 +0000 UTC m=+3001.735097303" watchObservedRunningTime="2026-11-04 01:48:36.795495468 +0000 UTC m=+3001.737344371" Nov 04 01:48:36.815277 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:36.815238 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-845d896dc4-gllnl" podStartSLOduration=2.05429786 podStartE2EDuration="2.815225615s" totalImagesPullingTime="760.927755ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:48:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:48:36.814147855 +0000 UTC m=+3001.755996771" watchObservedRunningTime="2026-11-04 01:48:36.815225615 +0000 UTC m=+3001.757074517" Nov 04 01:48:42.787300 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:42.787271 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-847bf86b74-76hwj" Nov 04 01:48:42.787719 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:42.787643 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-845d896dc4-gllnl" Nov 04 01:48:42.899672 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:42.899636 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-847bf86b74-76hwj"] Nov 04 01:48:42.899896 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:42.899849 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-847bf86b74-76hwj" podUID="e503a527-6c89-4264-bfd3-211f6d9d9dfd" containerName="maas-api" containerID="cri-o://b320754bba17dc83805a74c37c597cbcf78766a552034cc78154e6921f76c3f8" gracePeriod=30 Nov 04 01:48:43.158763 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:43.158741 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-847bf86b74-76hwj" Nov 04 01:48:43.315017 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:43.314988 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e503a527-6c89-4264-bfd3-211f6d9d9dfd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e503a527-6c89-4264-bfd3-211f6d9d9dfd-maas-api-tls\") pod \"e503a527-6c89-4264-bfd3-211f6d9d9dfd\" (UID: \"e503a527-6c89-4264-bfd3-211f6d9d9dfd\") " Nov 04 01:48:43.315183 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:43.315097 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e503a527-6c89-4264-bfd3-211f6d9d9dfd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e503a527-6c89-4264-bfd3-211f6d9d9dfd-openshift-service-ca\") pod \"e503a527-6c89-4264-bfd3-211f6d9d9dfd\" (UID: \"e503a527-6c89-4264-bfd3-211f6d9d9dfd\") " Nov 04 01:48:43.315183 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:43.315148 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e503a527-6c89-4264-bfd3-211f6d9d9dfd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e503a527-6c89-4264-bfd3-211f6d9d9dfd-metrics-certs\") pod \"e503a527-6c89-4264-bfd3-211f6d9d9dfd\" (UID: \"e503a527-6c89-4264-bfd3-211f6d9d9dfd\") " Nov 04 01:48:43.315258 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:43.315194 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e503a527-6c89-4264-bfd3-211f6d9d9dfd-kube-api-access-v8kdt\" (UniqueName: \"kubernetes.io/projected/e503a527-6c89-4264-bfd3-211f6d9d9dfd-kube-api-access-v8kdt\") pod \"e503a527-6c89-4264-bfd3-211f6d9d9dfd\" (UID: \"e503a527-6c89-4264-bfd3-211f6d9d9dfd\") " Nov 04 01:48:43.315491 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:43.315464 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e503a527-6c89-4264-bfd3-211f6d9d9dfd-openshift-service-ca" pod "e503a527-6c89-4264-bfd3-211f6d9d9dfd" (UID: "e503a527-6c89-4264-bfd3-211f6d9d9dfd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:48:43.315601 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:43.315574 7059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e503a527-6c89-4264-bfd3-211f6d9d9dfd-openshift-service-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:43.318079 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:43.318025 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e503a527-6c89-4264-bfd3-211f6d9d9dfd-metrics-certs" pod "e503a527-6c89-4264-bfd3-211f6d9d9dfd" (UID: "e503a527-6c89-4264-bfd3-211f6d9d9dfd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:48:43.318209 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:43.318120 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e503a527-6c89-4264-bfd3-211f6d9d9dfd-kube-api-access-v8kdt" pod "e503a527-6c89-4264-bfd3-211f6d9d9dfd" (UID: "e503a527-6c89-4264-bfd3-211f6d9d9dfd"). InnerVolumeSpecName "kube-api-access-v8kdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:48:43.318271 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:43.318230 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e503a527-6c89-4264-bfd3-211f6d9d9dfd-maas-api-tls" pod "e503a527-6c89-4264-bfd3-211f6d9d9dfd" (UID: "e503a527-6c89-4264-bfd3-211f6d9d9dfd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:48:43.416163 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:43.416103 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e503a527-6c89-4264-bfd3-211f6d9d9dfd-maas-api-tls\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:43.416163 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:43.416126 7059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e503a527-6c89-4264-bfd3-211f6d9d9dfd-metrics-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:43.416163 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:43.416137 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8kdt\" (UniqueName: \"kubernetes.io/projected/e503a527-6c89-4264-bfd3-211f6d9d9dfd-kube-api-access-v8kdt\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:43.821078 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:43.821037 7059 generic.go:356] "Generic (PLEG): container finished" podID="e503a527-6c89-4264-bfd3-211f6d9d9dfd" containerID="b320754bba17dc83805a74c37c597cbcf78766a552034cc78154e6921f76c3f8" exitCode=0 Nov 04 01:48:43.821511 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:43.821104 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-847bf86b74-76hwj" Nov 04 01:48:43.821511 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:43.821121 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-847bf86b74-76hwj" event={"ID":"e503a527-6c89-4264-bfd3-211f6d9d9dfd","Type":"ContainerDied","Data":"b320754bba17dc83805a74c37c597cbcf78766a552034cc78154e6921f76c3f8"} Nov 04 01:48:43.821511 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:43.821189 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-847bf86b74-76hwj" event={"ID":"e503a527-6c89-4264-bfd3-211f6d9d9dfd","Type":"ContainerDied","Data":"edfdd36a6c066a7b35a4a1c7b50587dd610f973fe6fe14bda471a0f5c7f83ebf"} Nov 04 01:48:43.821511 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:43.821211 7059 scope.go:122] "RemoveContainer" containerID="b320754bba17dc83805a74c37c597cbcf78766a552034cc78154e6921f76c3f8" Nov 04 01:48:43.831171 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:43.831156 7059 scope.go:122] "RemoveContainer" containerID="b320754bba17dc83805a74c37c597cbcf78766a552034cc78154e6921f76c3f8" Nov 04 01:48:43.831426 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:48:43.831405 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b320754bba17dc83805a74c37c597cbcf78766a552034cc78154e6921f76c3f8\": container with ID starting with b320754bba17dc83805a74c37c597cbcf78766a552034cc78154e6921f76c3f8 not found: ID does not exist" containerID="b320754bba17dc83805a74c37c597cbcf78766a552034cc78154e6921f76c3f8" Nov 04 01:48:43.831480 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:43.831436 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b320754bba17dc83805a74c37c597cbcf78766a552034cc78154e6921f76c3f8"} err="failed to get container status \"b320754bba17dc83805a74c37c597cbcf78766a552034cc78154e6921f76c3f8\": rpc error: code = NotFound desc = could not find container \"b320754bba17dc83805a74c37c597cbcf78766a552034cc78154e6921f76c3f8\": container with ID starting with b320754bba17dc83805a74c37c597cbcf78766a552034cc78154e6921f76c3f8 not found: ID does not exist" Nov 04 01:48:43.848508 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:43.848483 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-847bf86b74-76hwj"] Nov 04 01:48:43.853650 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:43.853627 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-847bf86b74-76hwj"] Nov 04 01:48:45.176874 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:45.176834 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e503a527-6c89-4264-bfd3-211f6d9d9dfd" path="/var/lib/kubelet/pods/e503a527-6c89-4264-bfd3-211f6d9d9dfd/volumes" Nov 04 01:48:46.231399 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:46.231361 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc"] Nov 04 01:48:46.232109 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:46.232073 7059 cpu_manager.go:414] "containerMap: removing container" podUID="e503a527-6c89-4264-bfd3-211f6d9d9dfd" containerName="maas-api" Nov 04 01:48:46.232109 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:46.232091 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e503a527-6c89-4264-bfd3-211f6d9d9dfd" containerName="maas-api" Nov 04 01:48:46.232262 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:46.232244 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="e503a527-6c89-4264-bfd3-211f6d9d9dfd" containerName="maas-api" Nov 04 01:48:46.238191 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:46.238165 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc" Nov 04 01:48:46.243520 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:46.243490 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d76a5805-f959-48e6-a451-02431e18ccbf-metrics-certs\") pod \"maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc\" (UID: \"d76a5805-f959-48e6-a451-02431e18ccbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc" Nov 04 01:48:46.243733 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:46.243709 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zjg7\" (UniqueName: \"kubernetes.io/projected/d76a5805-f959-48e6-a451-02431e18ccbf-kube-api-access-4zjg7\") pod \"maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc\" (UID: \"d76a5805-f959-48e6-a451-02431e18ccbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc" Nov 04 01:48:46.243893 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:46.243865 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d76a5805-f959-48e6-a451-02431e18ccbf-openshift-service-ca\") pod \"maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc\" (UID: \"d76a5805-f959-48e6-a451-02431e18ccbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc" Nov 04 01:48:46.244048 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:46.243995 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d76a5805-f959-48e6-a451-02431e18ccbf-maas-api-tls\") pod \"maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc\" (UID: \"d76a5805-f959-48e6-a451-02431e18ccbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc" Nov 04 01:48:46.246999 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:46.246973 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc"] Nov 04 01:48:46.281582 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:46.281549 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b1e8de2d-serving-cert\"" Nov 04 01:48:46.344773 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:46.344738 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d76a5805-f959-48e6-a451-02431e18ccbf-maas-api-tls\") pod \"maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc\" (UID: \"d76a5805-f959-48e6-a451-02431e18ccbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc" Nov 04 01:48:46.344958 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:46.344825 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d76a5805-f959-48e6-a451-02431e18ccbf-metrics-certs\") pod \"maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc\" (UID: \"d76a5805-f959-48e6-a451-02431e18ccbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc" Nov 04 01:48:46.344958 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:46.344847 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zjg7\" (UniqueName: \"kubernetes.io/projected/d76a5805-f959-48e6-a451-02431e18ccbf-kube-api-access-4zjg7\") pod \"maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc\" (UID: \"d76a5805-f959-48e6-a451-02431e18ccbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc" Nov 04 01:48:46.344958 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:46.344870 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d76a5805-f959-48e6-a451-02431e18ccbf-openshift-service-ca\") pod \"maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc\" (UID: \"d76a5805-f959-48e6-a451-02431e18ccbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc" Nov 04 01:48:46.344958 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:48:46.344895 7059 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b1e8de2d-serving-cert: secret "maas-api-e2e-ait-b1e8de2d-serving-cert" not found Nov 04 01:48:46.345131 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:48:46.344987 7059 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d76a5805-f959-48e6-a451-02431e18ccbf-maas-api-tls podName:d76a5805-f959-48e6-a451-02431e18ccbf nodeName:}" failed. No retries permitted until 2026-11-04 01:48:46.844970891 +0000 UTC m=+3011.786819789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d76a5805-f959-48e6-a451-02431e18ccbf-maas-api-tls") pod "maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc" (UID: "d76a5805-f959-48e6-a451-02431e18ccbf") : secret "maas-api-e2e-ait-b1e8de2d-serving-cert" not found Nov 04 01:48:46.345584 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:46.345560 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d76a5805-f959-48e6-a451-02431e18ccbf-openshift-service-ca\") pod \"maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc\" (UID: \"d76a5805-f959-48e6-a451-02431e18ccbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc" Nov 04 01:48:46.347620 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:46.347597 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d76a5805-f959-48e6-a451-02431e18ccbf-metrics-certs\") pod \"maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc\" (UID: \"d76a5805-f959-48e6-a451-02431e18ccbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc" Nov 04 01:48:46.357512 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:46.357490 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zjg7\" (UniqueName: \"kubernetes.io/projected/d76a5805-f959-48e6-a451-02431e18ccbf-kube-api-access-4zjg7\") pod \"maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc\" (UID: \"d76a5805-f959-48e6-a451-02431e18ccbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc" Nov 04 01:48:46.849225 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:46.849192 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d76a5805-f959-48e6-a451-02431e18ccbf-maas-api-tls\") pod \"maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc\" (UID: \"d76a5805-f959-48e6-a451-02431e18ccbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc" Nov 04 01:48:46.852225 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:46.852195 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d76a5805-f959-48e6-a451-02431e18ccbf-maas-api-tls\") pod \"maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc\" (UID: \"d76a5805-f959-48e6-a451-02431e18ccbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc" Nov 04 01:48:46.857013 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:46.856990 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc" Nov 04 01:48:47.175235 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:48:47.175208 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd76a5805_f959_48e6_a451_02431e18ccbf.slice/crio-6785d4e91f953dbedf953b3458a4f3628742b39cbb2641ee887bb107347f4ee5 WatchSource:0}: Error finding container 6785d4e91f953dbedf953b3458a4f3628742b39cbb2641ee887bb107347f4ee5: Status 404 returned error can't find the container with id 6785d4e91f953dbedf953b3458a4f3628742b39cbb2641ee887bb107347f4ee5 Nov 04 01:48:47.177496 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:47.177466 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc"] Nov 04 01:48:47.851358 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:47.851295 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc" event={"ID":"d76a5805-f959-48e6-a451-02431e18ccbf","Type":"ContainerStarted","Data":"ad7650438693b8dfa7511b763e11c176bc321e2eaf7dc433bf0b18ed62216ee6"} Nov 04 01:48:47.851892 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:47.851859 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc" event={"ID":"d76a5805-f959-48e6-a451-02431e18ccbf","Type":"ContainerStarted","Data":"6785d4e91f953dbedf953b3458a4f3628742b39cbb2641ee887bb107347f4ee5"} Nov 04 01:48:47.852164 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:47.852146 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc" Nov 04 01:48:47.873023 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:47.872976 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc" podStartSLOduration=1.3043623850000001 podStartE2EDuration="1.872964273s" totalImagesPullingTime="568.601888ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:48:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:48:47.871162485 +0000 UTC m=+3012.813011387" watchObservedRunningTime="2026-11-04 01:48:47.872964273 +0000 UTC m=+3012.814813175" Nov 04 01:48:53.863991 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:48:53.863962 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc" Nov 04 01:49:06.479153 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:49:06.479112 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:49:10.650542 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:10.650503 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc"] Nov 04 01:49:10.651090 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:10.650717 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc" podUID="d76a5805-f959-48e6-a451-02431e18ccbf" containerName="maas-api" containerID="cri-o://ad7650438693b8dfa7511b763e11c176bc321e2eaf7dc433bf0b18ed62216ee6" gracePeriod=30 Nov 04 01:49:10.917832 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:10.917775 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc" Nov 04 01:49:10.993035 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:10.992997 7059 generic.go:356] "Generic (PLEG): container finished" podID="d76a5805-f959-48e6-a451-02431e18ccbf" containerID="ad7650438693b8dfa7511b763e11c176bc321e2eaf7dc433bf0b18ed62216ee6" exitCode=0 Nov 04 01:49:10.993237 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:10.993054 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc" Nov 04 01:49:10.993237 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:10.993069 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc" event={"ID":"d76a5805-f959-48e6-a451-02431e18ccbf","Type":"ContainerDied","Data":"ad7650438693b8dfa7511b763e11c176bc321e2eaf7dc433bf0b18ed62216ee6"} Nov 04 01:49:10.993237 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:10.993113 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc" event={"ID":"d76a5805-f959-48e6-a451-02431e18ccbf","Type":"ContainerDied","Data":"6785d4e91f953dbedf953b3458a4f3628742b39cbb2641ee887bb107347f4ee5"} Nov 04 01:49:10.993237 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:10.993136 7059 scope.go:122] "RemoveContainer" containerID="ad7650438693b8dfa7511b763e11c176bc321e2eaf7dc433bf0b18ed62216ee6" Nov 04 01:49:11.001386 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:11.001332 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d76a5805-f959-48e6-a451-02431e18ccbf-kube-api-access-4zjg7\" (UniqueName: \"kubernetes.io/projected/d76a5805-f959-48e6-a451-02431e18ccbf-kube-api-access-4zjg7\") pod \"d76a5805-f959-48e6-a451-02431e18ccbf\" (UID: \"d76a5805-f959-48e6-a451-02431e18ccbf\") " Nov 04 01:49:11.001501 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:11.001448 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d76a5805-f959-48e6-a451-02431e18ccbf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d76a5805-f959-48e6-a451-02431e18ccbf-openshift-service-ca\") pod \"d76a5805-f959-48e6-a451-02431e18ccbf\" (UID: \"d76a5805-f959-48e6-a451-02431e18ccbf\") " Nov 04 01:49:11.001600 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:11.001578 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d76a5805-f959-48e6-a451-02431e18ccbf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d76a5805-f959-48e6-a451-02431e18ccbf-metrics-certs\") pod \"d76a5805-f959-48e6-a451-02431e18ccbf\" (UID: \"d76a5805-f959-48e6-a451-02431e18ccbf\") " Nov 04 01:49:11.001820 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:11.001798 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d76a5805-f959-48e6-a451-02431e18ccbf-openshift-service-ca" pod "d76a5805-f959-48e6-a451-02431e18ccbf" (UID: "d76a5805-f959-48e6-a451-02431e18ccbf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:49:11.001918 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:11.001882 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d76a5805-f959-48e6-a451-02431e18ccbf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d76a5805-f959-48e6-a451-02431e18ccbf-maas-api-tls\") pod \"d76a5805-f959-48e6-a451-02431e18ccbf\" (UID: \"d76a5805-f959-48e6-a451-02431e18ccbf\") " Nov 04 01:49:11.002402 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:11.002380 7059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d76a5805-f959-48e6-a451-02431e18ccbf-openshift-service-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:11.003283 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:11.003257 7059 scope.go:122] "RemoveContainer" containerID="ad7650438693b8dfa7511b763e11c176bc321e2eaf7dc433bf0b18ed62216ee6" Nov 04 01:49:11.003676 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:49:11.003646 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad7650438693b8dfa7511b763e11c176bc321e2eaf7dc433bf0b18ed62216ee6\": container with ID starting with ad7650438693b8dfa7511b763e11c176bc321e2eaf7dc433bf0b18ed62216ee6 not found: ID does not exist" containerID="ad7650438693b8dfa7511b763e11c176bc321e2eaf7dc433bf0b18ed62216ee6" Nov 04 01:49:11.003790 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:11.003684 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad7650438693b8dfa7511b763e11c176bc321e2eaf7dc433bf0b18ed62216ee6"} err="failed to get container status \"ad7650438693b8dfa7511b763e11c176bc321e2eaf7dc433bf0b18ed62216ee6\": rpc error: code = NotFound desc = could not find container \"ad7650438693b8dfa7511b763e11c176bc321e2eaf7dc433bf0b18ed62216ee6\": container with ID starting with ad7650438693b8dfa7511b763e11c176bc321e2eaf7dc433bf0b18ed62216ee6 not found: ID does not exist" Nov 04 01:49:11.003868 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:11.003804 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d76a5805-f959-48e6-a451-02431e18ccbf-kube-api-access-4zjg7" pod "d76a5805-f959-48e6-a451-02431e18ccbf" (UID: "d76a5805-f959-48e6-a451-02431e18ccbf"). InnerVolumeSpecName "kube-api-access-4zjg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:49:11.003959 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:11.003939 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d76a5805-f959-48e6-a451-02431e18ccbf-metrics-certs" pod "d76a5805-f959-48e6-a451-02431e18ccbf" (UID: "d76a5805-f959-48e6-a451-02431e18ccbf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:49:11.004217 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:11.004198 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d76a5805-f959-48e6-a451-02431e18ccbf-maas-api-tls" pod "d76a5805-f959-48e6-a451-02431e18ccbf" (UID: "d76a5805-f959-48e6-a451-02431e18ccbf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:49:11.103104 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:11.103073 7059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d76a5805-f959-48e6-a451-02431e18ccbf-metrics-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:11.103104 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:11.103098 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d76a5805-f959-48e6-a451-02431e18ccbf-maas-api-tls\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:11.103104 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:11.103108 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zjg7\" (UniqueName: \"kubernetes.io/projected/d76a5805-f959-48e6-a451-02431e18ccbf-kube-api-access-4zjg7\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:11.323557 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:11.323522 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc"] Nov 04 01:49:11.343576 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:11.343542 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b1e8de2d-76f5498dfd-q5dqc"] Nov 04 01:49:13.176393 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:13.176357 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d76a5805-f959-48e6-a451-02431e18ccbf" path="/var/lib/kubelet/pods/d76a5805-f959-48e6-a451-02431e18ccbf/volumes" Nov 04 01:49:17.952293 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:17.952260 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs"] Nov 04 01:49:17.952877 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:17.952856 7059 cpu_manager.go:414] "containerMap: removing container" podUID="d76a5805-f959-48e6-a451-02431e18ccbf" containerName="maas-api" Nov 04 01:49:17.953030 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:17.952879 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d76a5805-f959-48e6-a451-02431e18ccbf" containerName="maas-api" Nov 04 01:49:17.953030 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:17.953023 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="d76a5805-f959-48e6-a451-02431e18ccbf" containerName="maas-api" Nov 04 01:49:17.957973 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:17.957946 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs" Nov 04 01:49:17.971529 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:17.971500 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-442e8c-serving-cert\"" Nov 04 01:49:17.977490 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:17.975672 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs"] Nov 04 01:49:18.072800 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:18.072758 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k822s\" (UniqueName: \"kubernetes.io/projected/37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da-kube-api-access-k822s\") pod \"maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs\" (UID: \"37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs" Nov 04 01:49:18.072999 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:18.072877 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs\" (UID: \"37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs" Nov 04 01:49:18.072999 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:18.072940 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da-metrics-certs\") pod \"maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs\" (UID: \"37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs" Nov 04 01:49:18.073146 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:18.073123 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs\" (UID: \"37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs" Nov 04 01:49:18.173642 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:18.173611 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs\" (UID: \"37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs" Nov 04 01:49:18.173829 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:18.173686 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da-metrics-certs\") pod \"maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs\" (UID: \"37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs" Nov 04 01:49:18.173829 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:18.173794 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs\" (UID: \"37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs" Nov 04 01:49:18.173994 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:18.173839 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k822s\" (UniqueName: \"kubernetes.io/projected/37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da-kube-api-access-k822s\") pod \"maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs\" (UID: \"37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs" Nov 04 01:49:18.174358 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:18.174332 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs\" (UID: \"37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs" Nov 04 01:49:18.176350 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:18.176327 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs\" (UID: \"37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs" Nov 04 01:49:18.176445 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:18.176378 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da-metrics-certs\") pod \"maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs\" (UID: \"37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs" Nov 04 01:49:18.182747 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:18.182724 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k822s\" (UniqueName: \"kubernetes.io/projected/37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da-kube-api-access-k822s\") pod \"maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs\" (UID: \"37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs" Nov 04 01:49:18.275454 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:18.275373 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs" Nov 04 01:49:18.553801 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:18.553768 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs"] Nov 04 01:49:18.555369 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:49:18.555342 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37e7c8e6_9f01_4f9a_bd7d_eb9aef3da2da.slice/crio-b55239e7bf2c842beccf4cfc4977ca22e48514ad45f04867a7b62b2f6c8509c0 WatchSource:0}: Error finding container b55239e7bf2c842beccf4cfc4977ca22e48514ad45f04867a7b62b2f6c8509c0: Status 404 returned error can't find the container with id b55239e7bf2c842beccf4cfc4977ca22e48514ad45f04867a7b62b2f6c8509c0 Nov 04 01:49:19.046219 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:19.046179 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs" event={"ID":"37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da","Type":"ContainerStarted","Data":"b55239e7bf2c842beccf4cfc4977ca22e48514ad45f04867a7b62b2f6c8509c0"} Nov 04 01:49:20.053872 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:20.053837 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs" event={"ID":"37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da","Type":"ContainerStarted","Data":"cee66f1e4b3139e686541e2b453e6364aaeb748e681028ca2a9734c1329440f1"} Nov 04 01:49:20.054302 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:20.053972 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs" Nov 04 01:49:20.073995 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:20.073940 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs" podStartSLOduration=2.449525326 podStartE2EDuration="3.073911888s" totalImagesPullingTime="624.386562ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:49:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:49:20.072976112 +0000 UTC m=+3045.014825019" watchObservedRunningTime="2026-11-04 01:49:20.073911888 +0000 UTC m=+3045.015760788" Nov 04 01:49:26.065305 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:26.065277 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs" Nov 04 01:49:36.480530 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:49:36.480485 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:49:36.754338 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:36.754264 7059 scope.go:122] "RemoveContainer" containerID="0384ba7f3cccf057dc4e174e18c2879855bb1470a685858799278589b013644f" Nov 04 01:49:48.024024 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:48.023917 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wwhm9"] Nov 04 01:49:48.025973 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:48.025949 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wwhm9"] Nov 04 01:49:49.177128 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:49:49.177100 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92932abf-4280-4b07-80f6-5c4c6fea0f72" path="/var/lib/kubelet/pods/92932abf-4280-4b07-80f6-5c4c6fea0f72/volumes" Nov 04 01:50:06.481153 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:50:06.481112 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:50:36.479784 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:50:36.479746 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:50:58.728691 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:50:58.728658 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs"] Nov 04 01:50:58.729265 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:50:58.728887 7059 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs" podUID="37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da" containerName="maas-api" containerID="cri-o://cee66f1e4b3139e686541e2b453e6364aaeb748e681028ca2a9734c1329440f1" gracePeriod=30 Nov 04 01:50:58.994492 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:50:58.994440 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs" Nov 04 01:50:59.073346 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:50:59.073316 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da-kube-api-access-k822s\" (UniqueName: \"kubernetes.io/projected/37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da-kube-api-access-k822s\") pod \"37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da\" (UID: \"37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da\") " Nov 04 01:50:59.073535 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:50:59.073374 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da-metrics-certs\" (UniqueName: \"kubernetes.io/secret/37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da-metrics-certs\") pod \"37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da\" (UID: \"37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da\") " Nov 04 01:50:59.073535 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:50:59.073506 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da-openshift-service-ca\") pod \"37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da\" (UID: \"37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da\") " Nov 04 01:50:59.073535 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:50:59.073525 7059 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da-maas-api-tls\") pod \"37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da\" (UID: \"37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da\") " Nov 04 01:50:59.073890 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:50:59.073862 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da-openshift-service-ca" pod "37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da" (UID: "37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:50:59.075757 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:50:59.075726 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da-metrics-certs" pod "37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da" (UID: "37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:50:59.075856 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:50:59.075805 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da-maas-api-tls" pod "37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da" (UID: "37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:50:59.075856 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:50:59.075833 7059 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da-kube-api-access-k822s" pod "37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da" (UID: "37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da"). InnerVolumeSpecName "kube-api-access-k822s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:50:59.174912 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:50:59.174886 7059 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k822s\" (UniqueName: \"kubernetes.io/projected/37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da-kube-api-access-k822s\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:50:59.174912 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:50:59.174911 7059 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da-metrics-certs\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:50:59.175095 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:50:59.174936 7059 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da-openshift-service-ca\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:50:59.175095 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:50:59.174949 7059 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da-maas-api-tls\") on node \"ip-10-0-128-129.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:50:59.638218 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:50:59.638187 7059 generic.go:356] "Generic (PLEG): container finished" podID="37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da" containerID="cee66f1e4b3139e686541e2b453e6364aaeb748e681028ca2a9734c1329440f1" exitCode=0 Nov 04 01:50:59.638392 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:50:59.638232 7059 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs" Nov 04 01:50:59.638392 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:50:59.638239 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs" event={"ID":"37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da","Type":"ContainerDied","Data":"cee66f1e4b3139e686541e2b453e6364aaeb748e681028ca2a9734c1329440f1"} Nov 04 01:50:59.638392 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:50:59.638280 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs" event={"ID":"37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da","Type":"ContainerDied","Data":"b55239e7bf2c842beccf4cfc4977ca22e48514ad45f04867a7b62b2f6c8509c0"} Nov 04 01:50:59.638392 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:50:59.638300 7059 scope.go:122] "RemoveContainer" containerID="cee66f1e4b3139e686541e2b453e6364aaeb748e681028ca2a9734c1329440f1" Nov 04 01:50:59.647726 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:50:59.647703 7059 scope.go:122] "RemoveContainer" containerID="cee66f1e4b3139e686541e2b453e6364aaeb748e681028ca2a9734c1329440f1" Nov 04 01:50:59.647992 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:50:59.647973 7059 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cee66f1e4b3139e686541e2b453e6364aaeb748e681028ca2a9734c1329440f1\": container with ID starting with cee66f1e4b3139e686541e2b453e6364aaeb748e681028ca2a9734c1329440f1 not found: ID does not exist" containerID="cee66f1e4b3139e686541e2b453e6364aaeb748e681028ca2a9734c1329440f1" Nov 04 01:50:59.648074 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:50:59.648007 7059 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cee66f1e4b3139e686541e2b453e6364aaeb748e681028ca2a9734c1329440f1"} err="failed to get container status \"cee66f1e4b3139e686541e2b453e6364aaeb748e681028ca2a9734c1329440f1\": rpc error: code = NotFound desc = could not find container \"cee66f1e4b3139e686541e2b453e6364aaeb748e681028ca2a9734c1329440f1\": container with ID starting with cee66f1e4b3139e686541e2b453e6364aaeb748e681028ca2a9734c1329440f1 not found: ID does not exist" Nov 04 01:50:59.655994 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:50:59.655974 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs"] Nov 04 01:50:59.660113 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:50:59.660090 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-442e8c-9857d8f8-nj7fs"] Nov 04 01:51:01.176084 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:51:01.176039 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da" path="/var/lib/kubelet/pods/37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da/volumes" Nov 04 01:51:06.481779 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:51:06.481740 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:51:33.954161 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:51:33.954018 7059 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-7rw6b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:51:33.954963 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:51:33.954899 7059 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-7rw6b" podUID="994ce0d7-9fe2-494e-a915-94fe06da822e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:51:36.479887 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:51:36.479850 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:51:36.793092 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:51:36.793015 7059 scope.go:122] "RemoveContainer" containerID="c1f0d2e90e68e6f454ad641d53eaa11243045d8d4aa37530944f5f5a4c8b5176" Nov 04 01:51:36.806990 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:51:36.806966 7059 scope.go:122] "RemoveContainer" containerID="ea8dbc242ad7177a58426e8e76e7a38591e2e6290066de8b1d550788866f3a69" Nov 04 01:52:06.479009 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:52:06.478967 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:52:08.886329 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:08.886289 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6556489bf6-qrnzz_b967836c-ea0e-4864-aee6-d76adbaa0656/manager/0.log" Nov 04 01:52:09.088106 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:09.088076 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-4xgpz_0a4d968b-87e6-43b5-b788-5ed8ce9da490/manager/0.log" Nov 04 01:52:09.740522 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:09.740491 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8498f949d-j6mmc_fc16e4aa-a8ba-42f2-8918-8e8c3ebcb78b/manager/0.log" Nov 04 01:52:09.969265 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:09.969236 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-wlxsc_73ab959e-6d9d-47fd-9e3b-aba4bca8451f/server/0.log" Nov 04 01:52:10.157960 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:10.157868 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-k6ljb_98a21413-61fe-4fc7-afc0-1984fd885d6a/manager/0.log" Nov 04 01:52:10.377098 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:10.377066 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-h4j6x_c37edcff-4510-449e-a03e-ff4be0250b41/proxy/0.log" Nov 04 01:52:11.346620 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:11.346590 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-845d896dc4-gllnl_93d9777a-32e4-4e1d-9231-78f201654998/maas-api/0.log" Nov 04 01:52:11.894166 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:11.894132 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2ced87dd-45r56_19c1d045-b17f-4a30-abbf-473456346eb7/revoke-keys/0.log" Nov 04 01:52:12.069571 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:12.069544 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-fp6dv_574bfa6e-f2d8-4283-bde1-56857a57c3f0/revoke-keys/0.log" Nov 04 01:52:12.250991 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:12.250964 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-z8s8k_8d93f92c-5a11-4a6e-9424-c5f01e53d280/revoke-keys/0.log" Nov 04 01:52:12.770607 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:12.770576 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-lchwg_31791d64-921a-4126-8c34-6ea469f60e13/postgres/0.log" Nov 04 01:52:14.831188 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:14.831160 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-b9mcd_eb98db7f-8c90-4e73-ba5d-4d7617d69cd2/manager/0.log" Nov 04 01:52:15.217168 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:15.217142 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-9zvzx_80b36f39-30fe-4f0e-bfc4-7945c0f4ac97/kuadrant-console-plugin/0.log" Nov 04 01:52:17.514256 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:17.514227 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85448bfccc-jwqpc_104eb9c5-8190-47a2-b0ce-ead26831913c/kube-auth-proxy/0.log" Nov 04 01:52:18.033939 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:18.033892 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-7rw6b_994ce0d7-9fe2-494e-a915-94fe06da822e/istio-proxy/0.log" Nov 04 01:52:18.416280 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:18.416201 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-bprgn_5e65c0d9-95d9-43a6-b77f-bd41b3fd6383/istio-proxy/0.log" Nov 04 01:52:18.623886 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:18.623860 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-ddd7x_26881d61-360e-43bb-83f8-3122e0026e41/istio-proxy/0.log" Nov 04 01:52:19.522693 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:19.522671 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5_100334ea-096e-40b5-9804-41e5fbfac86f/istio-proxy/0.log" Nov 04 01:52:19.688239 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:19.688213 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-qrllg_07b7d302-ff11-4c35-a56a-040c692c85f9/payload-pre-processing/0.log" Nov 04 01:52:19.863732 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:19.863671 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-ch84f_9a4edef8-b6bd-4c7e-a371-678894caee48/payload-processing/0.log" Nov 04 01:52:21.312402 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:21.312373 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-tl4kv_c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd/main/0.log" Nov 04 01:52:21.490576 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:21.490548 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-9clfp_7e6a52c4-c655-43d6-8a0e-200e8523e315/main/0.log" Nov 04 01:52:21.859802 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:21.859772 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7_23ea0f8f-58d8-43aa-b25b-f09ea8321e21/main/0.log" Nov 04 01:52:22.426088 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:22.426056 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm_abeae128-5b8c-4d5e-9592-f64e9c6bd4b5/main/0.log" Nov 04 01:52:32.615510 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:32.615479 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6556489bf6-qrnzz_b967836c-ea0e-4864-aee6-d76adbaa0656/manager/0.log" Nov 04 01:52:32.810075 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:32.810033 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-4xgpz_0a4d968b-87e6-43b5-b788-5ed8ce9da490/manager/0.log" Nov 04 01:52:33.406310 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:33.406276 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8498f949d-j6mmc_fc16e4aa-a8ba-42f2-8918-8e8c3ebcb78b/manager/0.log" Nov 04 01:52:33.600436 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:33.600373 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-wlxsc_73ab959e-6d9d-47fd-9e3b-aba4bca8451f/server/0.log" Nov 04 01:52:33.768846 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:33.768814 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-k6ljb_98a21413-61fe-4fc7-afc0-1984fd885d6a/manager/0.log" Nov 04 01:52:33.981828 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:33.981804 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-h4j6x_c37edcff-4510-449e-a03e-ff4be0250b41/proxy/0.log" Nov 04 01:52:34.529432 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:34.529402 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-845d896dc4-gllnl_93d9777a-32e4-4e1d-9231-78f201654998/maas-api/0.log" Nov 04 01:52:35.095643 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:35.095614 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2ced87dd-45r56_19c1d045-b17f-4a30-abbf-473456346eb7/revoke-keys/0.log" Nov 04 01:52:35.272590 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:35.272561 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-fp6dv_574bfa6e-f2d8-4283-bde1-56857a57c3f0/revoke-keys/0.log" Nov 04 01:52:35.459628 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:35.459607 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-z8s8k_8d93f92c-5a11-4a6e-9424-c5f01e53d280/revoke-keys/0.log" Nov 04 01:52:36.008236 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:36.008208 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-lchwg_31791d64-921a-4126-8c34-6ea469f60e13/postgres/0.log" Nov 04 01:52:36.480722 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:52:36.480686 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:52:37.393613 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:37.393585 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-b9mcd_eb98db7f-8c90-4e73-ba5d-4d7617d69cd2/manager/0.log" Nov 04 01:52:37.779719 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:37.779687 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-9zvzx_80b36f39-30fe-4f0e-bfc4-7945c0f4ac97/kuadrant-console-plugin/0.log" Nov 04 01:52:39.503972 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:39.503942 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85448bfccc-jwqpc_104eb9c5-8190-47a2-b0ce-ead26831913c/kube-auth-proxy/0.log" Nov 04 01:52:40.081610 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:40.081581 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-7rw6b_994ce0d7-9fe2-494e-a915-94fe06da822e/istio-proxy/0.log" Nov 04 01:52:40.495273 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:40.495236 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-bprgn_5e65c0d9-95d9-43a6-b77f-bd41b3fd6383/istio-proxy/0.log" Nov 04 01:52:40.700952 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:40.700908 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-ddd7x_26881d61-360e-43bb-83f8-3122e0026e41/istio-proxy/0.log" Nov 04 01:52:41.669490 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:41.669464 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5_100334ea-096e-40b5-9804-41e5fbfac86f/istio-proxy/0.log" Nov 04 01:52:41.876074 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:41.876042 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-qrllg_07b7d302-ff11-4c35-a56a-040c692c85f9/payload-pre-processing/0.log" Nov 04 01:52:42.068486 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:42.068454 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-ch84f_9a4edef8-b6bd-4c7e-a371-678894caee48/payload-processing/0.log" Nov 04 01:52:43.015161 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:43.015136 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-tl4kv_c973cb5b-66ec-4ec2-9cb3-0b8faa77f6bd/main/0.log" Nov 04 01:52:43.198523 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:43.198492 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-9clfp_7e6a52c4-c655-43d6-8a0e-200e8523e315/main/0.log" Nov 04 01:52:43.564130 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:43.564097 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8gj4z7_23ea0f8f-58d8-43aa-b25b-f09ea8321e21/main/0.log" Nov 04 01:52:44.124008 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:44.123977 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-2fsvm_abeae128-5b8c-4d5e-9592-f64e9c6bd4b5/main/0.log" Nov 04 01:52:54.023394 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:54.023319 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z8s8k"] Nov 04 01:52:54.027636 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:54.027609 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-z8s8k"] Nov 04 01:52:55.176267 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:52:55.176236 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d93f92c-5a11-4a6e-9424-c5f01e53d280" path="/var/lib/kubelet/pods/8d93f92c-5a11-4a6e-9424-c5f01e53d280/volumes" Nov 04 01:53:06.480829 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:53:06.480791 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:53:08.024848 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:53:08.024812 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fp6dv"] Nov 04 01:53:08.026873 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:53:08.026850 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fp6dv"] Nov 04 01:53:09.176805 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:53:09.176770 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="574bfa6e-f2d8-4283-bde1-56857a57c3f0" path="/var/lib/kubelet/pods/574bfa6e-f2d8-4283-bde1-56857a57c3f0/volumes" Nov 04 01:53:11.760084 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:53:11.760052 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8498f949d-j6mmc_fc16e4aa-a8ba-42f2-8918-8e8c3ebcb78b/manager/0.log" Nov 04 01:53:12.033348 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:53:12.033274 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8498f949d-j6mmc_fc16e4aa-a8ba-42f2-8918-8e8c3ebcb78b/manager/0.log" Nov 04 01:53:12.767513 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:53:12.767484 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-ch84f_9a4edef8-b6bd-4c7e-a371-678894caee48/payload-processing/0.log" Nov 04 01:53:12.943070 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:53:12.943041 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-ch84f_9a4edef8-b6bd-4c7e-a371-678894caee48/payload-processing/0.log" Nov 04 01:53:26.041949 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:53:26.041903 7059 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ced87dd-45r56"] Nov 04 01:53:26.047208 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:53:26.047180 7059 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ced87dd-45r56"] Nov 04 01:53:27.176610 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:53:27.176560 7059 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19c1d045-b17f-4a30-abbf-473456346eb7" path="/var/lib/kubelet/pods/19c1d045-b17f-4a30-abbf-473456346eb7/volumes" Nov 04 01:53:36.479356 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:53:36.479317 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:53:48.018799 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:53:48.018765 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8498f949d-j6mmc_fc16e4aa-a8ba-42f2-8918-8e8c3ebcb78b/manager/0.log" Nov 04 01:53:48.267327 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:53:48.267294 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8498f949d-j6mmc_fc16e4aa-a8ba-42f2-8918-8e8c3ebcb78b/manager/0.log" Nov 04 01:53:48.966763 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:53:48.966728 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-ch84f_9a4edef8-b6bd-4c7e-a371-678894caee48/payload-processing/0.log" Nov 04 01:53:49.138694 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:53:49.138664 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-ch84f_9a4edef8-b6bd-4c7e-a371-678894caee48/payload-processing/0.log" Nov 04 01:53:53.416780 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:53:53.416743 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2x7kl_bac70f0f-4585-42fe-abba-916ac41fcc62/global-pull-secret-syncer/0.log" Nov 04 01:53:53.713469 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:53:53.713435 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nb955_de6f6762-66c6-42c0-bf0d-7966483a040d/konnectivity-agent/0.log" Nov 04 01:53:53.822672 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:53:53.822642 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-129.us-east-2.compute.internal_14ca05fc22f96fd5090f819c755a4a44/haproxy/0.log" Nov 04 01:54:01.537657 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:01.537627 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-b9mcd_eb98db7f-8c90-4e73-ba5d-4d7617d69cd2/manager/0.log" Nov 04 01:54:01.618217 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:01.618192 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-9zvzx_80b36f39-30fe-4f0e-bfc4-7945c0f4ac97/kuadrant-console-plugin/0.log" Nov 04 01:54:03.308746 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.308711 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-8wz25_95638663-fffb-41a6-8056-c1f4293a3fb0/kube-state-metrics/0.log" Nov 04 01:54:03.340166 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.340139 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-8wz25_95638663-fffb-41a6-8056-c1f4293a3fb0/kube-rbac-proxy-main/0.log" Nov 04 01:54:03.364745 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.364723 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-8wz25_95638663-fffb-41a6-8056-c1f4293a3fb0/kube-rbac-proxy-self/0.log" Nov 04 01:54:03.397388 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.397369 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-845fbdd675-9rvps_559e1de2-7e21-4aeb-916f-7a75e0c75841/metrics-server/0.log" Nov 04 01:54:03.425205 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.425182 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-4z274_5bcf3164-e6e8-4323-aad0-ee8adbe4a77d/monitoring-plugin/0.log" Nov 04 01:54:03.454199 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.454179 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mhb4l_5522ff12-fed1-4411-96b0-1cceb6f0b634/node-exporter/0.log" Nov 04 01:54:03.478816 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.478798 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mhb4l_5522ff12-fed1-4411-96b0-1cceb6f0b634/kube-rbac-proxy/0.log" Nov 04 01:54:03.512320 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.512301 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mhb4l_5522ff12-fed1-4411-96b0-1cceb6f0b634/init-textfile/0.log" Nov 04 01:54:03.680985 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.680956 7059 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pplq5/perf-node-gather-daemonset-vbqzb"] Nov 04 01:54:03.681442 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.681428 7059 cpu_manager.go:414] "containerMap: removing container" podUID="37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da" containerName="maas-api" Nov 04 01:54:03.681498 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.681444 7059 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da" containerName="maas-api" Nov 04 01:54:03.681579 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.681568 7059 memory_manager.go:365] "RemoveStaleState removing state" podUID="37e7c8e6-9f01-4f9a-bd7d-eb9aef3da2da" containerName="maas-api" Nov 04 01:54:03.685051 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.685028 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-vbqzb" Nov 04 01:54:03.694277 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.694237 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pplq5\"/\"openshift-service-ca.crt\"" Nov 04 01:54:03.694277 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.694253 7059 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pplq5\"/\"kube-root-ca.crt\"" Nov 04 01:54:03.694449 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.694333 7059 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pplq5\"/\"default-dockercfg-lhzcx\"" Nov 04 01:54:03.695838 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.695815 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pplq5/perf-node-gather-daemonset-vbqzb"] Nov 04 01:54:03.754381 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.754355 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-dl2dc_1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84/kube-rbac-proxy-main/0.log" Nov 04 01:54:03.766067 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.766045 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d728dc0a-7b94-4ec1-994c-c1583dd4457e-sys\") pod \"perf-node-gather-daemonset-vbqzb\" (UID: \"d728dc0a-7b94-4ec1-994c-c1583dd4457e\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-vbqzb" Nov 04 01:54:03.766166 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.766088 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d728dc0a-7b94-4ec1-994c-c1583dd4457e-lib-modules\") pod \"perf-node-gather-daemonset-vbqzb\" (UID: \"d728dc0a-7b94-4ec1-994c-c1583dd4457e\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-vbqzb" Nov 04 01:54:03.766227 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.766209 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwt42\" (UniqueName: \"kubernetes.io/projected/d728dc0a-7b94-4ec1-994c-c1583dd4457e-kube-api-access-dwt42\") pod \"perf-node-gather-daemonset-vbqzb\" (UID: \"d728dc0a-7b94-4ec1-994c-c1583dd4457e\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-vbqzb" Nov 04 01:54:03.766274 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.766257 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d728dc0a-7b94-4ec1-994c-c1583dd4457e-podres\") pod \"perf-node-gather-daemonset-vbqzb\" (UID: \"d728dc0a-7b94-4ec1-994c-c1583dd4457e\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-vbqzb" Nov 04 01:54:03.766346 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.766334 7059 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d728dc0a-7b94-4ec1-994c-c1583dd4457e-proc\") pod \"perf-node-gather-daemonset-vbqzb\" (UID: \"d728dc0a-7b94-4ec1-994c-c1583dd4457e\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-vbqzb" Nov 04 01:54:03.780649 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.780628 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-dl2dc_1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84/kube-rbac-proxy-self/0.log" Nov 04 01:54:03.811182 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.811163 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-dl2dc_1fdfa1d2-7870-4cf2-aaf3-ea7a919bfb84/openshift-state-metrics/0.log" Nov 04 01:54:03.850276 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.850256 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0b7c4ca5-64c2-43d7-9963-90b35205391b/prometheus/0.log" Nov 04 01:54:03.867191 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.867169 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d728dc0a-7b94-4ec1-994c-c1583dd4457e-sys\") pod \"perf-node-gather-daemonset-vbqzb\" (UID: \"d728dc0a-7b94-4ec1-994c-c1583dd4457e\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-vbqzb" Nov 04 01:54:03.867288 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.867207 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d728dc0a-7b94-4ec1-994c-c1583dd4457e-lib-modules\") pod \"perf-node-gather-daemonset-vbqzb\" (UID: \"d728dc0a-7b94-4ec1-994c-c1583dd4457e\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-vbqzb" Nov 04 01:54:03.867288 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.867251 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwt42\" (UniqueName: \"kubernetes.io/projected/d728dc0a-7b94-4ec1-994c-c1583dd4457e-kube-api-access-dwt42\") pod \"perf-node-gather-daemonset-vbqzb\" (UID: \"d728dc0a-7b94-4ec1-994c-c1583dd4457e\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-vbqzb" Nov 04 01:54:03.867288 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.867277 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d728dc0a-7b94-4ec1-994c-c1583dd4457e-podres\") pod \"perf-node-gather-daemonset-vbqzb\" (UID: \"d728dc0a-7b94-4ec1-994c-c1583dd4457e\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-vbqzb" Nov 04 01:54:03.867413 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.867297 7059 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d728dc0a-7b94-4ec1-994c-c1583dd4457e-proc\") pod \"perf-node-gather-daemonset-vbqzb\" (UID: \"d728dc0a-7b94-4ec1-994c-c1583dd4457e\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-vbqzb" Nov 04 01:54:03.867413 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.867317 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d728dc0a-7b94-4ec1-994c-c1583dd4457e-lib-modules\") pod \"perf-node-gather-daemonset-vbqzb\" (UID: \"d728dc0a-7b94-4ec1-994c-c1583dd4457e\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-vbqzb" Nov 04 01:54:03.867413 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.867275 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d728dc0a-7b94-4ec1-994c-c1583dd4457e-sys\") pod \"perf-node-gather-daemonset-vbqzb\" (UID: \"d728dc0a-7b94-4ec1-994c-c1583dd4457e\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-vbqzb" Nov 04 01:54:03.867413 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.867375 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d728dc0a-7b94-4ec1-994c-c1583dd4457e-podres\") pod \"perf-node-gather-daemonset-vbqzb\" (UID: \"d728dc0a-7b94-4ec1-994c-c1583dd4457e\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-vbqzb" Nov 04 01:54:03.867560 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.867418 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d728dc0a-7b94-4ec1-994c-c1583dd4457e-proc\") pod \"perf-node-gather-daemonset-vbqzb\" (UID: \"d728dc0a-7b94-4ec1-994c-c1583dd4457e\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-vbqzb" Nov 04 01:54:03.873718 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.873701 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0b7c4ca5-64c2-43d7-9963-90b35205391b/config-reloader/0.log" Nov 04 01:54:03.876827 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.876806 7059 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwt42\" (UniqueName: \"kubernetes.io/projected/d728dc0a-7b94-4ec1-994c-c1583dd4457e-kube-api-access-dwt42\") pod \"perf-node-gather-daemonset-vbqzb\" (UID: \"d728dc0a-7b94-4ec1-994c-c1583dd4457e\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-vbqzb" Nov 04 01:54:03.898385 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.898363 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0b7c4ca5-64c2-43d7-9963-90b35205391b/thanos-sidecar/0.log" Nov 04 01:54:03.927723 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.927703 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0b7c4ca5-64c2-43d7-9963-90b35205391b/kube-rbac-proxy-web/0.log" Nov 04 01:54:03.952860 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.952814 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0b7c4ca5-64c2-43d7-9963-90b35205391b/kube-rbac-proxy/0.log" Nov 04 01:54:03.981249 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.981226 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0b7c4ca5-64c2-43d7-9963-90b35205391b/kube-rbac-proxy-thanos/0.log" Nov 04 01:54:03.997884 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:03.997864 7059 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-vbqzb" Nov 04 01:54:04.016917 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:04.016895 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0b7c4ca5-64c2-43d7-9963-90b35205391b/init-config-reloader/0.log" Nov 04 01:54:04.267575 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:04.267504 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d5d5b78fd-wjqgg_6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a/thanos-query/0.log" Nov 04 01:54:04.275091 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:04.275065 7059 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pplq5/perf-node-gather-daemonset-vbqzb"] Nov 04 01:54:04.276190 ip-10-0-128-129 kubenswrapper[7059]: W1104 01:54:04.276162 7059 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd728dc0a_7b94_4ec1_994c_c1583dd4457e.slice/crio-5964145b656a891f624fe0d5c5cc74faf613437dc675395d4c48b2aa408e2a9f WatchSource:0}: Error finding container 5964145b656a891f624fe0d5c5cc74faf613437dc675395d4c48b2aa408e2a9f: Status 404 returned error can't find the container with id 5964145b656a891f624fe0d5c5cc74faf613437dc675395d4c48b2aa408e2a9f Nov 04 01:54:04.293088 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:04.293064 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d5d5b78fd-wjqgg_6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a/kube-rbac-proxy-web/0.log" Nov 04 01:54:04.321325 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:04.321302 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d5d5b78fd-wjqgg_6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a/kube-rbac-proxy/0.log" Nov 04 01:54:04.348613 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:04.348592 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d5d5b78fd-wjqgg_6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a/prom-label-proxy/0.log" Nov 04 01:54:04.379980 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:04.379954 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d5d5b78fd-wjqgg_6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a/kube-rbac-proxy-rules/0.log" Nov 04 01:54:04.404858 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:04.404837 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d5d5b78fd-wjqgg_6c5b0800-f9a2-4ce6-a3d8-5459cbf9a70a/kube-rbac-proxy-metrics/0.log" Nov 04 01:54:04.786672 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:04.786640 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-vbqzb" event={"ID":"d728dc0a-7b94-4ec1-994c-c1583dd4457e","Type":"ContainerStarted","Data":"4b84f89377d7afad0bd9fb332a4f3af2893926716d2e350149071fb2901945a4"} Nov 04 01:54:04.786841 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:04.786699 7059 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-vbqzb" event={"ID":"d728dc0a-7b94-4ec1-994c-c1583dd4457e","Type":"ContainerStarted","Data":"5964145b656a891f624fe0d5c5cc74faf613437dc675395d4c48b2aa408e2a9f"} Nov 04 01:54:04.786841 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:04.786764 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-vbqzb" Nov 04 01:54:04.811078 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:04.811039 7059 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-vbqzb" podStartSLOduration=1.811027078 podStartE2EDuration="1.811027078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:54:04.808960771 +0000 UTC m=+3329.750809674" watchObservedRunningTime="2026-11-04 01:54:04.811027078 +0000 UTC m=+3329.752876022" Nov 04 01:54:06.479586 ip-10-0-128-129 kubenswrapper[7059]: E1104 01:54:06.479548 7059 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7jm5h" containerName="extractor" Nov 04 01:54:06.721340 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:06.721320 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-65dcbb7874-lwgl8_df880d85-2da0-40fc-a106-22af2ad25cd7/console/0.log" Nov 04 01:54:08.573407 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:08.573379 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hhndm_d08dca6d-7563-4d2c-ae92-e7ffd4beda06/dns/0.log" Nov 04 01:54:08.596591 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:08.596568 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hhndm_d08dca6d-7563-4d2c-ae92-e7ffd4beda06/kube-rbac-proxy/0.log" Nov 04 01:54:08.749711 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:08.749686 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h4s87_388103ca-b2f5-42b2-96ce-737f1ef00dc1/dns-node-resolver/0.log" Nov 04 01:54:09.476417 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:09.476384 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z4snb_61e0c4ee-edf6-4938-96c0-c79812e00840/node-ca/0.log" Nov 04 01:54:10.803268 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:10.803241 7059 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-vbqzb" Nov 04 01:54:11.664108 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:11.664073 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85448bfccc-jwqpc_104eb9c5-8190-47a2-b0ce-ead26831913c/kube-auth-proxy/0.log" Nov 04 01:54:11.758183 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:11.758158 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-7rw6b_994ce0d7-9fe2-494e-a915-94fe06da822e/istio-proxy/0.log" Nov 04 01:54:11.828171 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:11.828144 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-bprgn_5e65c0d9-95d9-43a6-b77f-bd41b3fd6383/istio-proxy/0.log" Nov 04 01:54:11.864021 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:11.863999 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-ddd7x_26881d61-360e-43bb-83f8-3122e0026e41/istio-proxy/0.log" Nov 04 01:54:12.037716 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:12.037691 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5_100334ea-096e-40b5-9804-41e5fbfac86f/istio-proxy/0.log" Nov 04 01:54:12.065606 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:12.065582 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-qrllg_07b7d302-ff11-4c35-a56a-040c692c85f9/payload-pre-processing/0.log" Nov 04 01:54:12.092558 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:12.092538 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-ch84f_9a4edef8-b6bd-4c7e-a371-678894caee48/payload-processing/0.log" Nov 04 01:54:14.392491 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:14.392458 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cv4w2_d90c5697-ccdd-4845-8972-dbbd952dcc87/serve-healthcheck-canary/0.log" Nov 04 01:54:15.097489 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:15.097462 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7jm5h_59e9f4f9-590f-43b0-999d-1fad31779d16/kube-rbac-proxy/0.log" Nov 04 01:54:15.121412 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:15.121389 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7jm5h_59e9f4f9-590f-43b0-999d-1fad31779d16/exporter/0.log" Nov 04 01:54:15.144576 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:15.144558 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7jm5h_59e9f4f9-590f-43b0-999d-1fad31779d16/extractor/0.log" Nov 04 01:54:15.528533 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:15.528509 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85448bfccc-jwqpc_104eb9c5-8190-47a2-b0ce-ead26831913c/kube-auth-proxy/0.log" Nov 04 01:54:15.591322 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:15.591303 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-7rw6b_994ce0d7-9fe2-494e-a915-94fe06da822e/istio-proxy/0.log" Nov 04 01:54:15.630396 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:15.630371 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-bprgn_5e65c0d9-95d9-43a6-b77f-bd41b3fd6383/istio-proxy/0.log" Nov 04 01:54:15.647045 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:15.647025 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-ddd7x_26881d61-360e-43bb-83f8-3122e0026e41/istio-proxy/0.log" Nov 04 01:54:15.749759 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:15.749735 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5_100334ea-096e-40b5-9804-41e5fbfac86f/istio-proxy/0.log" Nov 04 01:54:15.765064 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:15.765040 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-qrllg_07b7d302-ff11-4c35-a56a-040c692c85f9/payload-pre-processing/0.log" Nov 04 01:54:15.785109 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:15.785063 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-ch84f_9a4edef8-b6bd-4c7e-a371-678894caee48/payload-processing/0.log" Nov 04 01:54:18.068867 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:18.068841 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m_730d376a-4902-4852-b314-68767aad19cb/manager/0.log" Nov 04 01:54:18.088419 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:18.088394 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7d9c85cd8c-s9l4m_730d376a-4902-4852-b314-68767aad19cb/copy-manifests/0.log" Nov 04 01:54:19.200387 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:19.200353 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85448bfccc-jwqpc_104eb9c5-8190-47a2-b0ce-ead26831913c/kube-auth-proxy/0.log" Nov 04 01:54:19.274980 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:19.274955 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-7rw6b_994ce0d7-9fe2-494e-a915-94fe06da822e/istio-proxy/0.log" Nov 04 01:54:19.310268 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:19.310237 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-bprgn_5e65c0d9-95d9-43a6-b77f-bd41b3fd6383/istio-proxy/0.log" Nov 04 01:54:19.329265 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:19.329241 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-ddd7x_26881d61-360e-43bb-83f8-3122e0026e41/istio-proxy/0.log" Nov 04 01:54:19.413784 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:19.413761 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-operator-674cd59dc5-dgmxr_db25d5e4-2c15-426f-80dd-f0ecb20cb1bc/jobset-operator/0.log" Nov 04 01:54:19.421314 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:19.421294 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-5cf4665f7f-2sbw5_100334ea-096e-40b5-9804-41e5fbfac86f/istio-proxy/0.log" Nov 04 01:54:19.436715 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:19.436697 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-qrllg_07b7d302-ff11-4c35-a56a-040c692c85f9/payload-pre-processing/0.log" Nov 04 01:54:19.454008 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:19.453919 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-ch84f_9a4edef8-b6bd-4c7e-a371-678894caee48/payload-processing/0.log" Nov 04 01:54:20.144694 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:20.144668 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6556489bf6-qrnzz_b967836c-ea0e-4864-aee6-d76adbaa0656/manager/0.log" Nov 04 01:54:20.168539 ip-10-0-128-129 kubenswrapper[7059]: I1104 01:54:20.168515 7059 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-4xgpz_0a4d968b-87e6-43b5-b788-5ed8ce9da490/manager/0.log"